Workbox Plugin loads full HTML of JIRA comment, leads to GC loop of death on large comment

2015-08-04T13:06:54
ID ATLASSIAN:CONFSERVER-38699
Type atlassian
Reporter martin.sander
Modified 2018-10-11T09:03:53

Description

To reproduce:

  • start Confluence with GC logging enabled (optional, but helps)
  • Link Confluence and JIRA
  • create an issue in JIRA
  • watch it
  • add a large comment to the JIRA issue, e.g. paste a 7.7MB log file between {code} tags
  • open the workbox in Confluence (optional: in network tab of web developer tools)
  • check CPU usage of Confluence: it switches between one http thread using 100% CPU to render the comment and as many threads as the GC uses to collect the garbage that the thread produces GC log of Confluence: you will see frequent full garbage collections, all freeing big parts of the heap (in our case for example: Full GC every 5 seconds, heap 5G -> 1G, taking 2-5 seconds) if Confluence is not dead yet, check the network tab in the browser: {{/rest/mywork/latest/notification/nested?limit=30}} loads a huge file (examples in our case: 21MB, 43MB before gzip), as it loads the whole comment in html take a heap dump of Confluence just before a garbage collection. When analyzing in Eclipse Memory Analyzer, use [-keep_unreachable_objects|https://wiki.eclipse.org/MemoryAnalyzer/FAQ#How_to_analyse_unreachable_objects] *** you will find big char arrays/Strings containing the various parts of the comment, embedded in {{org.apache.xerces.dom.TextImpl}}, which suggests that the whole comment is parsed as html/xml and kept in memory at once
  • check the table {{AO_9412A1_AONOTIFICATION}} in the Confluence database. It contains several entries with a large {{DESCRIPTION}} field, even though only one large comment was added. It seems that one comment triggers new entries in this table again and again, which is also reflected in the workbox that says "9 comments on <ISSUEKEY>", although there is only one comment.

On a production system with a reasonable load, enough request queue up during the GC pauses to render the system unusable - garbage is produced faster than it can be collected.

Note: this has security implications as well, as any user that can comment on issues can DOS the system easily - they do not even require another user to watch the ticket, as they are notified of their own JIRA actions in Confluence, see CONF-30081.

Suggestions for mitigation: * JIRA should not allow comments that big * JIRA should not send the whole comment to Confluence * Confluence should stop processing the comment after a few lines, and show a "read more" link instead of the whole comment (like the JIRA activity stream does)