The forums are read only! All new discussions will need to take place on answers.atlassian.com. These forums will remain available as a read only resource. Some features have been disabled. See the Answers FAQ for more information.

Home » Atlassian Forums » Software » JIRA Development



Permlink Replies: 4 - Pages: 1 - Last Post: Feb 15, 2008 12:18 PM Last Post By: Matt Doar
Dylan Thurston
[JIRA-dev] AddAttachmentToIssue slow, and too mem/cpu intensive.
Posted: Jan 14, 2008 11:35 AM
Click to report abuse...   Click to reply to this thread Reply
Hello,
A little while back someone posted about how to attach files to
tickets using the soap interface. in perl you can use the following
code:

my $h_result = undef;
my $refh_attachment = undef;
if( open( ATTACHMENT_SEND, '<'t read from file handle for
\"".$filepath."\": " . $! . ".\n";
} else {
$refh_attachment = { 'name' => undef, 'contents' => []};
$refh_attachment->{'name'} = SOAP::Data->type(
'xsd:string' => basename( $filepath ));
push( @{ $refh_attachment->{'contents'}}, map{
SOAP::Data->type('xsd:byte' => $_ )} @{ $refar_byte });
}
} else {
warn "Can't set file handle to binary mode for
\"".$filepath . "\": " . $! . ".\n";
}
close( ATTACHMENT_SEND );
} else {
warn "Can't open file \"".$filepath."\" for reading: ".$!.".\n";
}
if( defined( $refh_attachment )) {
if( !defined( $h_result =
$self->{soap}->addAttachmentsToIssue( $self->{authToken},$issueid,
$refh_attachment->{ 'name' }, $refh_attachment->{ 'contents' })) ||
$h_result->fault()){
print "Unable to attach files to new issue
\"".$issueid."\" : ". ( defined( $h_result ) ?
$h_result->faultstring() : "unknown cause." )."\n";
} else {
#print "Added attachments to issue \"".$issueid."\".\n";
}
}

This works great, with one minor issue. It is way too slow to be
useable. It immediately pegs my cpu to the limit, and uses almost all
of the memory that it can. not to mention the fact that if the f ile
is larger than about 1 meg it is the program dies, and anything below
that limit takes several minutes to do. I've narrowed the inital
problem down to the line:

push( @{ $refh_attachment->{'contents'}}, map{
SOAP::Data->type('xsd:byte' => $_ )} @{ $refar_byte });

Which as far as i can tell goes through the unpacked array of bytes,
byte by byte and stuffs it into the attachment->{contents} as soap
bytes (which are much larger than regular bytes as they contain
definition data as well as the bytes. I was just hopeing someone might
be able to help me figure out how to attach files to an issue more
efficiently. There are times where i will need to attach a file that
is 20-50 megs, and the slowdown and cpu/mem use is just unacceptable.
I'm thinking that sending the data as a byte array might not be the
best way to transfer this data across a soap interface. I'm
considering patching the code on the soap plugin, to accept the data
as a base64 mime encoded string. rathar than a gigantic array of
bytes. If anyone else can think of a way of attaching files that
doesn't require patching the soap plugin i'd love to hear it. or if
you thinks base64 mime encoded strings are the best way to go i'd also
like to know what you think is a better route. Also has anyone else
ran into the same problem or am i doing something wrong?
-Dylan
_______________________________________________
Jira-developer mailing list
Jira-developer@lists.atlassian.com
To unsubscribe or change your options visit this page:
http://lists.atlassian.com/mailman/listinfo/jira-developer

Matt Doar

Posts: 445
Registered: 06/01/05
Re: [JIRA-dev] AddAttachmentToIssue slow, and too mem/cpu intensive.
Posted: Jan 14, 2008 3:17 PM    global.in_response_to.tooltip in response to: Dylan Thurston
Click to report abuse...   Click to reply to this thread Reply
Dylan Thurston
Re: [JIRA-dev] AddAttachmentToIssue slow, and too mem/cpu intensive.
Posted: Jan 14, 2008 7:50 PM    global.in_response_to.tooltip in response to: Matt Doar
Click to report abuse...   Click to reply to this thread Reply
In case anyone is interested. I patched the jira rpc plugin and added
a new function addBase64AttachmentToIssue, this new function accepts
strings instead of byte arrays. My new function is way more efficient
than the old one.The new function can actually handle 5-10 meg files
in a matter of seconds. I was unable to test files larger than that
because of a bug with perl's Soap::Lite. I'm preatty sure i know the
problem with the old attachment function is that it accepts byte
arrays, unfortunately from what i can tell when you create a byte
array in soap it encapsalates each individual byte with a lot of xml.
this means that every byte is probably ~30 times it's actual
size.generating that much xml and trying to send it to the soap server
this is likely the problem. the new function encapsulates the entire
base64 encode file in 1 single string xml tag. while encoding to
base64 does increase the size of the data, it seems like best solution
to make the binary data safe to send as a string. If anyone is
interested i can send you the patch, or the newly compiled plugin and
you're welcome test it out for yourself. Now i only need to find a
solution to my out of memory problem with soap::lite.I'll probably
test the new plugin out with python or c# tomorrow (if i get a chance)
and i'll let you know how big of a file i'm able to reasonably attach.
If anyone is interested in what i've done please let me know.
-Dylan

On Jan 14, 2008 2:17 PM, <forums@atlassian.com> wrote:
Take a look at

http://jira.atlassian.com/browse/JRA-11693

~Matt
--
Post by mdoar - online at:
http://forums.atlassian.com/thread.jspa?forumID=100&threadID=22534
_______________________________________________
Jira-developer mailing list
Jira-developer@lists.atlassian.com
To unsubscribe or change your options visit this page:
http://lists.atlassian.com/mailman/listinfo/jira-developer

_______________________________________________
Jira-developer mailing list
Jira-developer@lists.atlassian.com
To unsubscribe or change your options visit this page:
http://lists.atlassian.com/mailman/listinfo/jira-developer
Matt Doar

Posts: 445
Registered: 06/01/05
Re: [JIRA-dev] AddAttachmentToIssue slow, and too mem/cpu intensive.
Posted: Jan 15, 2008 12:29 PM    global.in_response_to.tooltip in response to: Dylan Thurston
Click to report abuse...   Click to reply to this thread Reply
Nice work! Yes, please post the patch or the whole new file or plugin, whatever works for you. I'd suggest adding it to the JIRA issue that I mentioned earlier.
Matt Doar

Posts: 445
Registered: 06/01/05
Re: [JIRA-dev] AddAttachmentToIssue slow, and too mem/cpu intensive.
Posted: Feb 15, 2008 12:18 PM    global.in_response_to.tooltip in response to: Dylan Thurston
Click to report abuse...   Click to reply to this thread Reply
Dylan,

Did you ever post that patch?

~Matt
Legend
Genius: 1000 - 10000 pts
Expert: 500 - 999 pts
Guide: 250 - 499 pts
Enthusiast: 50 - 249 pts
Newbie: 5 - 49 pts
Atlassian Employee
Atlassian Partner
Genius!
Helpful Answer (5 pts)
Correct Answer (10 pts)

Point your RSS reader here for a feed of the latest messages in all forums