Reporting Issues
Last modified: 20 April 2023If you experience problems running TeamCity and believe it's related to the software, please contact us with detailed description of the issue.
To fix a problem, we may need a wide range of information about your system as well as various logs. The section below explains how to collect such information.
Hangs and Thread Dumps
If you experience problems with the TeamCity server or agent (e.g. no responding or working too slow) we would appreciate a thread dump of the process taken in the moment of the slowness.
Determine what process is slow If there is an issue with the server or build sources checkout issue, you will need a thread dump of the web server process (e.g. Tomcat), if the issue is related only to a single build, you will need an agent thread dump. Please also consider investigating the system CPU load and find the process that is loading the CPU (if any). If it is not TeamCity-related process, that might need addressing not in the scope of TeamCity.
Server thread dump
You can take a thread dump of the TeamCity server right from the web UI (if the hanging is local and you can still open administration pages): go to the Administration | Server Configuration | Diagnostics tab and click the View server thread dump link to open the thread dump in a new browser window or Save Thread Dump button to save it to <TeamCity home>/logs
directory (where you can later download the files from "Server Logs"). A small hint: if web UI is not responsive try direct URL substituting your server URL.
If the server UI is not usable please use approaches described below. Please note that if you can take a thread dump without using TeamCity UI, that can be preferable since the thread dump taken from UI can lack some diagnostics information.
Agent thread dump
If you need agent thread dump, please note that TeamCity agent consists of two java
processes: launcher and agent itself. Agent is run by the launcher process. You will usually be interested in the agent (more nested one) and not the launcher process.
Thread dump taking approaches
To take a thread dump: Under Windows You have several options:
if you need server thread dump and the server is run from console, press Ctrl+Break in the console window (this will not work for agent, since it's console belongs to launcher process).
if JDK 1.6 is used, use
jstack <pid_of_java_process>
command (jstack is located in the "bin" directory of JDK installation).you can also use TeamCity-bundled thread dump tool (can be found in agent's plugins). Run the command:
<TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process>
If the hanging process is run as a service, taking the thread dump is more complicated and under some environments does not have a reliable approach. Recommended approach is to run the agent or server process from a console and use Ctrl+Break
approach above.
Alternatively, you can try to run a thread dumping tool using Microsoft PsExec. Use the following command line to get the dump:
psexec -u <user> -p <password> <full path to the util> <process pid> stacktrace <output file>
where: <user> and <password> – correspond to the same account that was used to run the hanging process as a service. If service uses "System", use "-s" switch instead of -u
and -p
. <full path to the util> – full path to the thread dumping util <process pid> – the process id of the hanging process <output file> – a full path to a file to save output to (should be accessible for the specified user).
run
jstack <pid_of_java_process>
orkill -3 <pid_of_java_process>
. In the latter case output will appear in<TeamCity server home>/logs/catalina.out
or<TeamCity agent home>/logs/error.log
.
You can also use third-party GUI tool: AdaptJ StackTrace Utility. It supports Windows, Linux or Mac OS X. Choose "Launch" on the page (if you have Java installed). When the application is started, select the process id via "Process > Select", then dump its thread dump with "Process > Thread Dump".
See also Server Performance section below.
Database-related slowdowns
When the server is slow it makes scene to check if slowness is caused by the database operations. The specific tools are database-specific.
MySQL
Along with server thread dump, please attach output of "show processlist;" SQL command executed in MySQL console. Much like the thread dumps it makes sense to execute the command when the slowness is occurring several times and send us the output. Also, MySQL can be set up to keep a log of long queries executed with the changes in my.ini
:
[mysqld]
...
log-slow-queries
long_query_time=15
The log can also be sent to us for analysis.
OutOfMemory Problems
If you experience problems with TeamCity "eating" too much memory (OutOfMemory errors), please do the following:
Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent)
If server is to blame, please check you have increased memory settings from default ones for using the server in production (see the section).
If you use x64 JVM, please consider using 32 bit JVM (instructions for server).
Try to increase the memory for the process via
'-Xmx'
JVM option, like -Xmx512m. If the error message is "java.lang.OutOfMemoryError: PermGen space", increase the value in-XX:MaxPermSize=250m
JVM option. The option needs to be passed to the process with problems:if it is the building process itself, use "JVM Command Line Parameters" settings in the build runner. e.g. Inspections builds may specifically need to increase the parameter;
refer to the corresponding documentation for TeamCity server or agent. See also Installing and Configuring the TeamCity Server;
If increasing memory size does not help, please get the memory dump, archive it and send it to us for further analysis. Please reduce the Xmx setting to normal before getting the dump (smaller snapshots are easier to analyze and easier to upload):
to get a memory dump (hprof file) automatically when an OutOfMemory error occurs, add the following JVM option (works for JDK 1.5.0_07+):
-XX:+HeapDumpOnOutOfMemoryError
. When OOM error occurs next time,java_xxx.hprof
file will be created in the process startup directory (<TeamCity home>/bin
or<TeamCity Agent home>/bin
);if for some reason the previous option does not suit, you can take the memory dump manually right from the web UI when the memory usage is at it's peak. Go to the Administration | Server Configuration | Diagnostics tab and click Dump Memory Snapshot.
See how to change JVM options for the server and for agents.
Logging Events logging
TeamCity server and agent create logs that can be used to investigate issues.
tip
Before reproducing the problem it makes sense to enable 'DEBUG' log level for TeamCity classes. On the server side, go to the Administration | Server Configuration | Diagnostics tab and select debug logging level. After that, DEBUG messages will go to
teamcity-*.log
files (read more).
For detailed information, please refer to the corresponding sections:
TeamCity Server Logs Viewing Build Agent Logs
Version Control Debug Logging vcs
In general, to debug VCS problems we need information for jetbrains.buildServer.VCS Log4j category. You can switch logging preset to "debug-vcs" in administration web UI and then retrieve logs/teamcity-vcs.log
log file.
Alternatively, you can change the Log4j configuration manually in <TeamCity home>\conf\teamcity-server-log4j.xml or <BuildAgent home>\conf\teamcity-agent-log4j.xml files to have fragment:
<category name="jetbrains.buildServer.VCS" additivity="false">
<appender-ref ref="ROLL.VCS"/>
<appender-ref ref="CONSOLE-ERROR"/>
<priority value="DEBUG"/>
</category>
If there are separate logging options for specific version controls, they are described below.
Subversion Integration Debug Logging
Use "debug-vcs" logging preset in administration web UI and then retrieve logs/teamcity-svn.log
log file.
Alternative manual approach: First, please enable generic VCS debug logging, as described above.
Uncomment SVN-related parts (SVN.LOG
appender and javasvn.output
category) of Log4j configuration file on server and on agent (if agent-side checkout is used). The log will be saved to the logs/teamcity-svn.log
file. Generic VCS log should be also taken from logs/teamcity-vcs.log
ClearCase
Uncomment Clearcase-related lines in the <TeamCity home>\conf\teamcity-server-log4j.xml
file. The log will be saved to logs\teamcity-clearcase.log
directory.
Patch Application Problems
In case server-side checkout is used, the "patch" that is passed from server to the agent can be retrieved by:
add property
agent.save.patch=true
to the build configuration properties: Administration->Edit configuration->Properties and requirement variables->Add new property...trigger the build.
Agent log will contain the line "Patch is saved to file ${file.name}" Get the file and provide it with the problem description.
Remote Run Problems
The changes that are sent form the IDE to the server on a remote run can be retrieved from server's .BuildServer\system\changes
directory. Locate the <change_number>.changes
file that corresponds to your change (you can pick the latest number available or deduce the from the URL of the change form the web UI). The file contains the patch in the binary form. Please provide it with the problem description.
Server Performance
If you experience degraded server performance and TeamCity server process is producing large CPU load, please take the CPU profiling snapshot and send it to us accompanied with the detailed description of what you were doing and what is your system setup. You can take the CPU profiling and memory snapshots by installing the server profiling plugin and following the instructions provided on the plugin page.
Here are some hints to get the best results from CPU profiling:
after server startup wait for some time to allow it to "warm up". This can take from 5 to 20 minutes depending on the data volume that TeamCity stores.
when CPU usage increase is found on the server, please try to indicate what actions cause the load
start CPU profiling and repeat the action several times (5 - 10)
capture the snapshot
archive the snapshot and send it to us including description of the actions that cause CPU load
Logging in TeamCity Visual Studio plugin
To capture logs from TeamCity Visual Studio plugin please do the following:
Close all instances of Microsoft Visual Studio.
Open
<username-profile-folder>\Local Settings\Temp\JetLogs
folder.Delete all of the files in this folder.
Restart Microsoft Visual Studio.
Open a solution.
Try to log in to TeamCity.
Close Microsoft Visual Studio.
Navigate back to the
<username-profile-folder>\Local Settings\Temp\JetLogs
folder. All of the files that were created are logs.
Logging in TeamCity Eclipse plugin
Available only since TeamCity 4.5
To enable tracing for the plugin, run Eclipse IDE with the -debug <filename>
program argument. The <filename>
portion of the program argument is a properties file containing key-value pairs. Name of each property corresponds to the plugin module and value is either 'true
' (to enable debug) or 'false'
. Here is an example of enabling most common tracing options:
jetbrains.teamcity.core/debug = true
jetbrains.teamcity.core/debug/communications = false
jetbrains.teamcity.core/debug/ui = true
jetbrains.teamcity.core/debug/vcs = true
jetbrains.teamcity.core/debug/vcs/detail = true
jetbrains.teamcity.core/debug/parser = true
jetbrains.teamcity.core/debug/platform = true
jetbrains.teamcity.core/debug/teamcity = true
jetbrains.teamcity.core/perfomance/vcs = true
jetbrains.teamcity.core/perfomance/teamcity = true
Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.
Sending Information to the Developers
Files under 5Mb in size can be attached right into the tracker issue (if you do not want the attachments to be publicly accessible, limit viewing the attachment to "teamcity-developers" user group only).
If the file is over 5 Mb, you can upload the archived files via https://uploads.jetbrains.com.
You can also send small files via email: teamcity-feedback@jetbrains.com Please do not forget to mention your TeamCity version and environment