CDash

General

Our CDash gives an overview of our nightly builds as well as all continuous integration test run by Jenkins.

HowTo

File Locations

What	Location
Configuration	/web/cdash.seqan.de/cdash/CDash/config/config.local.php
Error log	/web/cdash.seqan.de/log/error.log
General log	/web/cdash.seqan.de/cdash/CDash/log/cdash.log
Storage of asynchronously submitted builds	/web/cdash.seqan.de/cdash/CDash/backup
PHP scripts, some callable from browser	/web/cdash.seqan.de/cdash/CDash/public /web/cdash.seqan.de/cdash/CDash/public/ajax

Asynchronous Submission

General

If the submissions are handled sequential, each submission is also processed immediately. This has the advantage that no temporary files need to be stored. The main drawback is that other submitting servers have to wait until the processing is done which may lead to timeouts.
When parallel submissions are enabled, the submitted XML files are stored under the backup directory and are processed independently of the submission process - submitters don't have to wait until the previous submission is processed.
To do so, CDash has to send a curl request to itself in order to trigger submission processing; this will require some configuration on our part.

Check Status

You can check the status at https://cdash.seqan.de/monitor.php after logging into your CDash account.

Needed Configuration Parameters

$CDASH_ASYNCHRONOUS_SUBMISSION = true; Enables parallel submission.
$CDASH_ASYNC_WORKERS = 1; Even though we use MySQL, which should support parallel access, setting this to anything higher than 1 results in stalled processing.

We also need these carefully chosen parameters:

$CDASH_CURL_REQUEST_LOCALHOST='0';
$CDASH_CURL_LOCALHOST_PREFIX='';
$CDASH_BASE_URL='';
$CDASH_SERVER_NAME = '';
$CDASH_SERVER_PORT = '';

Not setting these parameters results in a failed request to process the submissions, i.e. the server can report to CDash, but the reports are never processed, resulting in an empty overview in CDash.

Manually Trigger Submission Processing

This should only be done when the status check reveals unprocessed submissions and no submissions are currently processed.

Open in your browser (click to reveal)

https://cdash.seqan.de/ajax/processsubmissions.php?projectid=4

If this failed because some handle couldn't be acquired, call (click to reveal)

https://cdash.seqan.de/ajax/processsubmissions.php?projectid=4&force=true

Leave tab open until done.

Troubleshooting

No submissions being processed

If the monitor shows several hundred submission to be processed (Pending Submissions) and there is no progress, try to clear the contents of the backup folder and run the submission processing as described above.

MySQL has gone away

It can also happen that the error MySQL has gone away occurs (see log files). For our setup this usually happens when the database gets so big that the queries made by CDash cannot be completed in time (50 seconds limit). Last time the DB got over 4 GiB big and contained more than 35 million entries in a single table (e.g. SELECT COUNT(*) FROM test).
This will also stall the builds and there are two ways to fix this:

The easy way

• Log into your CDash account
• Visit https://cdash.seqan.de/upgrade.php and click Cleanup CDash

The long way

If the Cleanup CDash also times out (many MySQL has gone away errors are shown on the web page), we have to do it manually. Connect to the SQL DB in an SQL manager of your choice (See the config for server information).

Here is the query that CDash will try to run on the SQL DB (click to reveal)

DELETE FROM banner WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM blockbuild WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM build WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM buildgroup WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM labelemail WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM project2repositories WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM dailyupdate WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM projectrobot WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM submission WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM subproject WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM coveragefilepriority WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM test WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM user2project WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM userstatistics WHERE projectid NOT IN (SELECT id AS projectid FROM project);
DELETE FROM build2configure WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM build2note WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM build2test WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM buildemail WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM builderror WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM builderrordiff WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM buildfailure WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM buildinformation WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM buildnote WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM buildtesttime WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM configure WHERE id NOT IN (SELECT configureid AS id FROM build2configure);
DELETE FROM configureerror WHERE configureid NOT IN (SELECT id AS configureid FROM configure);
DELETE FROM configureerrordiff WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM coverage WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM coveragefilelog WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM coveragesummary WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM coveragesummarydiff WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM dynamicanalysis WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM label2build WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM subproject2build WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM summaryemail WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM testdiff WHERE buildid NOT IN (SELECT id AS buildid FROM build);
DELETE FROM dynamicanalysisdefect WHERE dynamicanalysisid NOT IN (SELECT id AS dynamicanalysisid FROM dynamicanalysis);
DELETE FROM subproject2subproject WHERE subprojectid NOT IN (SELECT id AS subprojectid FROM subproject);
DELETE FROM dailyupdatefile WHERE dailyupdateid NOT IN (SELECT id AS dailyupdateid FROM dailyupdate);
DELETE FROM coveragefile WHERE id NOT IN (SELECT fileid AS id FROM coverage);
DELETE FROM coveragefile2user WHERE fileid NOT IN (SELECT id AS fileid FROM coveragefile);
DELETE FROM dailyupdatefile WHERE dailyupdateid NOT IN (SELECT id AS dailyupdateid FROM dailyupdate);
DELETE FROM test2image WHERE testid NOT IN (SELECT id AS testid FROM test);
DELETE FROM testmeasurement WHERE testid NOT IN (SELECT id AS testid FROM test);
DELETE FROM label2test WHERE testid NOT IN (SELECT id AS testid FROM test);

Click here to reveal how to easily track the sent SQL queries

Locate the pdo.php file inside the include directory of the CDash directory and find the functionpdo_query

function pdo_query($query, $link_identifier = null, $log_error = true)
{
    $cur_pdo = get_link_identifier($link_identifier)->getPdo($log_error);
    if ($cur_pdo === false) {
        return false;
    } else {
        return $cur_pdo->query($query);
    }
}

and edit it to add an echo $query;

function pdo_query($query, $link_identifier = null, $log_error = true)
{
    echo $query;
    $cur_pdo = get_link_identifier($link_identifier)->getPdo($log_error);
    if ($cur_pdo === false) {
        return false;
    } else {
        return $cur_pdo->query($query);
    }
}

If you cannot edit the file, you may be able to move it, copy it and edit the copy.
Don't forget to revert the changes after you are done.
After changing pdo_query, you can execute Cleanup CDash as described in the previous section. After a while all sent SQL queries will be printed at the top of the screen.

While some of these will work, others will time out - these queries then need to be replaced by a TRUNCATE.

Here is what worked last time (click to reveal)

TRUNCATE `seqan_cdash`.`label2test`;
TRUNCATE `seqan_cdash`.`testmeasurement`;
TRUNCATE `seqan_cdash`.`testdiff`;
TRUNCATE `seqan_cdash`.`summaryemail`;
TRUNCATE `seqan_cdash`.`subproject2build`;
TRUNCATE `seqan_cdash`.`label2build`;
TRUNCATE `seqan_cdash`.`dynamicanalysis`;
TRUNCATE `seqan_cdash`.`coveragesummarydiff`;
TRUNCATE `seqan_cdash`.`coveragesummary`;
TRUNCATE `seqan_cdash`.`coveragefilelog`;
TRUNCATE `seqan_cdash`.`coverage`;
TRUNCATE `seqan_cdash`.`configureerrordiff`;
TRUNCATE `seqan_cdash`.`configureerror`;
TRUNCATE `seqan_cdash`.`configure`;
TRUNCATE `seqan_cdash`.`buildtesttime`;
TRUNCATE `seqan_cdash`.`buildnote`;
TRUNCATE `seqan_cdash`.`buildinformation`;
TRUNCATE `seqan_cdash`.`buildfailure`;
TRUNCATE `seqan_cdash`.`builderrordiff`;
TRUNCATE `seqan_cdash`.`builderror`;
TRUNCATE `seqan_cdash`.`build2test`;