Write a Stream to a Stream?


What’s up yall? I got this crazy programming requirement that has been killing me for the past couple of days. I need a way of serializing my streams. Now before you go cross eyed and start wagging your finger at me let me explain.

  • I know a stream already represents serialized data.
  • What I’m asking may sound like, “How do I wet my water?” but bear with me because I’m not as dumb as my picture looks.
  • I know that when I get into situations like this that require a lot of extra coding and thinking that I’m most likely doing something else wrong but in this case I really think I’m onto something big.

Here’s the big idea. I want an interface that I can thro into my code that relieves me from the burdens/limitations of Java serialization. The interface must work with streams, readers, and writers and must do so remotely without letting the caller know or care about remoting issues. The idea is really a generic repository abstraction that allows me to easily throw junk in it and read the same junk back later on. I’ve implemented a Filesystem version of the interface that maps all stuff thrown at it to the file system and I can easily bang out an in-memory version of the same thing that uses a HashMap or the equivalent. I now want to spread remoting over top as a layer of icing on my repository cake. (Mmmmm… cake…!) I’d like to be able to do such things as chain a RemoteHttpRepository to a RemoteSocketRepository to a RepositoryFarm containing a FileSystemRepository and an WriteOnlyArchivedRepository for backup purposes. I’d like to be able to change my mind later on and snap in a DBRepository inplace of the RemoteSocketRepository in the same example.

So now we come back to my initial Repository interface. It’s a really cool thing that allows you to pas arbitrary stuff into it and get that stuff back out of it. That stuff can be anything from a File object to a Reader or an inputStream. Passing Readers or streams causes it to slurp the contents of the stream and record it all for later retrieval. That brings me back to my above dillemma, what happens when the repository is remote? I have some initial code I threw together to try to address the problem but it’s getting big and ugly. Sure the idea started simple (as most off the cuff brilliant ideas we all have start off) but it quickly exploded into an array of classes, interfaces, and unit tests on my hard drive. That’s one of the downsides to intentional programming with a powerful editor like Idea. You can hammer away for a minute or so and create a few dozen files orchestrating a complicated object and dependancy graph without even knowing it. Now I’m stuck, partly because I thought it would be a lot simpler than it is and partly because with all the development I’ve done I only see a small piece of the picture as implemented. The RemoteRepository spun off an entire abstract remoting package complete with its own set of drama. While it’s being developed specifically for my repository needs it’s showing a whole other barrel of complexity that I’m avoiding by focusing on my specific needs. That scares me because I don’t want to limit its usage to repository access but I’m not ready to swallow the entire elephant either.

In short, I feel like I’m in over my head. If you’re browsing my site and of reasonable intellect I ask, no, I plead for your insight. What’s the best way to stream a stream? Am I crazy for doing what I’m doing? Is there something obvious that I’m missing? I’ve been Googling around and I can’t find much help. I don’t have anywhere else to turn to, so I ask you my loyal reader for guidance. Hit me off with the right stuff…

2 thoughts on “Write a Stream to a Stream?

  1. Can’t see none of the source code from here…😉

    I’m interested to understand where you’re going with this. You start to lose me about here:

    Passing Readers or streams causes it to slurp the contents of the stream and record it all for later retrieval.

    I didn’t quite understand what the purpose of the API is. There are always going to be difficulties with certain objects: e.g. what does it mean to want to persist a Socket? Where I’m going with this is:

    What types of problems suffer burdens/limitations of Java serialization and involve needing to store and retrieve streams?

  2. The big idea is this. I have an abstract repository interface that I want to be able to throw things at. It looks something like this:

    interface Repositroy {
    /**
    *Saves something in the repository uusing the name as the id.
    **/
    void save(String name, Object artifact);
    /**
    *load an artifact from the repository uusing the name as the id.
    **/
    Object load(String name);
    /**
    *load an artifact from the repository uusing the name as the id.
    **/
    void delete(String name);
    /**
    * list everything saved in the repository
    **/
    List list();
    }

    I want to be able to call the save method with an input stream and have the implementation write the contents of the stream to storage. I also want to be able call load and get an input stream back. I have an implementation that uses the file system for storage and it works with all of the above methods. It holds a general contract that if a simple data type like a String or a number is “saved” then that simple type is returned on load. If a stream, or a reader is passed to the save method then I get a stream back on load. The challenge come when I attempt to write a remote implementation of the Repository. Thats when I need a clever way of “saving” a stream to a remote repository. That’s when I need to pass a Stream over a stream. The general idea is to use a protocol that determines the type passed to the interface. When an input stream is given it needs to request an output stream from the remote server and pipe the data from input to output. I hope that’s a little more clear. It’s late and I’m drinking a yuengling so I can’t go into too much more detail. Maybe I’ll post a followup article on the topic to be more explanative.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s