David Hirschfield wrote:
> Thanks for the great response.
>> Yeah, by "safe" I mean that it's all happening on an intranet with no
> chance of malicious individuals getting access to the stream of data.
>> The chunks are arbitrary collections of python objects. I'm wrapping
> them up a little, but I don't know much about the actual formal makeup
> of the data, other than it pickles successfully.
>> Are there any existing python modules that do the equivalent of pickling
> on arbitrary python data, but do it a lot faster? I wasn't aware of any
> that are as easy to use as pickle, or don't require implementing them
> myself, which is not something I have time for.
>Marshal may achieve what you want, but on a more limited range of
datatypes than pickle.
regards
Steve
> Thanks again,
> -Dave
>> Steve Holden wrote:
>>>David Hirschfield wrote:
>>>>>>>I have a pair of programs which trade python data back and forth by
>>>pickling up lists of objects on one side (using
>>>pickle.HIGHEST_PROTOCOL), and sending that data over a TCP socket
>>>connection to the receiver, who unpickles the data and uses it.
>>>>>>So far this has been working fine, but I now need a way of separating
>>>multiple chunks of pickled binary data in the stream being sent back and
>>>forth.
>>>>>>Questions:
>>>>>>Is it safe to do what I'm doing? I didn't think there was anything
>>>fundamentally wrong with sending binary pickled data, especially in the
>>>closed, safe environment these programs operate under...but maybe I'm
>>>making a poor assumption?
>>>>>>>>>>>If there's no chance of malevolent attackers modifying the data stream
>>then you can safely ignore the otherwise dire consequences of unpickling
>>arbitrary chunks of data.
>>>>>>>>>I was going to separate the chunks of pickled data with some well-formed
>>>string, but couldn't that string potentially randomly appear in the
>>>pickled data? Do I just pick an extremely
>>>unlikely-to-be-randomly-generated string as the separator? Is there some
>>>string that will definitely NEVER show up in pickled binary data?
>>>>>>>>>>>I presumed each chunk was of a know structure. Couldn't you just lead of
>>with a pickled integer saying how many chunks follow?
>>>>>>>>>I thought about base64 encoding the data, and then decoding on the
>>>opposite side (like what xmlrpclib does), but that turns out to be a
>>>very expensive operation, which I want to avoid, speed is of the essence
>>>in this situation.
>>>>>>>>>>>Yes, base64 stuffs three bytes into four (six bits per byte) giving you
>>a 33% overhead. Having said that, pickle isn't all that efficient a
>>representation because it's designed to be portable. If you are using
>>machines of the same type there are almost certainly faster binary
>>encodings.
>>>>>>>>>Is there a reliable way to determine the byte count of some pickled
>>>binary data? Can I rely on len(<pickled data>) == bytes?
>>>>>>>>>>>Yes, since pickle returns a string of bytes, not a Unicode object.
>>>>If bandwidth really is becoming a limitation you might want to consider
>>uses of the struct module to represent things more compactly (but this
>>may be too difficult if the objects being exchanged are at all complex).
>>>>regards
>> Steve
>>>>>> --
> Presenting:
> mediocre nebula.
>
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden
