Results 1 to 7 of 7

Thread: db4o

  1. #1
    Join Date
    Oct 2006
    Location
    Bergen, Norway
    Posts
    365

    Default db4o

    Hi!

    Anyone using this technology? I'm thinking of using it for storing simulation results, that means whole object graphs of data, and it is so much data that our current way of storing these simulations increases the database size with 20GB pr year.

    It would be great if db4o would simplify data retrival, make it retrive data faster and at the same time reduce the incremental growth and overall size of our databse.

    Cheers,
    Steinar.

  2. #2
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,
    Yes, though I'm not the best person to comment on it. I've pinged Drik Louwers who has done some nice work on the db40 for .NET extension project to comment more on this thread.
    Cheers,
    Mark

  3. #3
    Join Date
    Oct 2008
    Posts
    12

    Default

    Hi,

    Depending on your exact requirements this would be a good idea. If your requirements are:

    - Easy storage of whole object graphs
    - Easy retrieval of whole object graphs
    - No references needed between all object graphs

    First of all the positive points:

    You can store a whole object graph by just using:

    #Store(myObjectGraphRoot)

    You can retrieve that whole object graph (or rather an IList implementor with the root object as it's sole element) very easily, based on some member variable of the root object or a db4o generated id.

    You can let db4o immediately activate the whole graph or use db4o's transparent activation to have them populated "lazily".

    The restriction:

    However, db4o does not like huge database files. It's possible but not advisable. If you have no need for queries that need info from any object graph and object graphs don't need to reference eachother you can create a new database file once the current one has grown beyond a certain size. You could even hatch your own strategy to open an objectcontainer on the right file to be able to retrieve random object graphs.

    Feel free to contact me if you have additional questions.

    Best,

    Dirk Louwers

  4. #4
    Join Date
    Oct 2006
    Location
    Bergen, Norway
    Posts
    365

    Default

    Hi!

    Thanks for the reply. Seems like this technology suits my needs, as far as I know, each object graph stands on it's own and there is no cross reference. Each object graph is a snapshot of a simulation at a specific point in time. For querying I would be interested in getting back the simulation root object closest to a date time value.

    I'll play with it and report back how it goes.

    Cheers,
    Steinar.

  5. #5
    Join Date
    Oct 2006
    Location
    Bergen, Norway
    Posts
    365

    Default

    I spent an hour testing db4o and ravenDb as well just to see how these scale in a dataintensive scenario, as such scenarios are most interesting to me. The result is that both object oriented databases do not perform well with dataseries. The test scenario is an oversimplified object graph containing a single class and a series of data in a list. More specific a simble class Valuta containing historical hourly valuta values for the last 27,5 years. In comparison I've worked with a big object graph containing data series around 800 000 values (records) and saving/fetching this massive amount of data from a relational database using NHibernate took around 2 seconds (of course I had spent some time optimizing NHibernate and the DB to achieve that result).

    And here are the results,

    Performance test of db4o saving and loading 240 000 valuta value instances in milliseconds:
    Saving operation:
    Time used inserting data: 15478
    Press 'Enter' to continue....

    Loading operation:
    Time used executing query: 10802
    Press 'Enter' to continue....

    And the code:
    Code:
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using Db4objects.Db4o;
    
    
    namespace Db4Objects
    {
        class Program
        {
            static void Main(string[] args)
            {
                Console.WriteLine("Press 'Enter' to read from database");
                ConsoleKeyInfo keyPressed = Console.ReadKey();
                var store = Db4oEmbedded.OpenFile("db4oTest.Yap");
                if (keyPressed.Key == ConsoleKey.Enter) ReadFromDB(store);
                else InsertIntoDB(store);
    
    
                
                store.Dispose();
            }
    
    
            private static void ReadFromDB(IObjectContainer store)
            {
                TimeSpan timeUsed;
                
                    DateTime start = DateTime.Now;
                    var v = store.Query<Valuta>().SingleOrDefault(x => x.Name == "GBP");//Where(x => x.Name.Equals("GBP"))
                        //Advanced.LuceneQuery<Valuta>("ValutaIndex").Where("Name:GBP").WaitForNonStaleResults().Single();
                    timeUsed = DateTime.Now - start;
    
    
                    Console.WriteLine("Valuta: {0} | {1}", v.Id, v.Name);
                    foreach (var val in v.Values)
                    {
                        Console.WriteLine("Val: {0} at {1}", val.Value, val.Time);
                    }
    
    
                Console.WriteLine("Time used executing query: "+timeUsed.TotalMilliseconds);
                Console.WriteLine("Press 'Enter' to continue....");
                Console.ReadLine();
            }
    
    
            private static void InsertIntoDB(IObjectContainer store)
            {
                DateTime today = DateTime.Now.Date;
                int numberOfValues = 240000;
                ValutaValue[] values = new ValutaValue[numberOfValues];
                for (int i = 0; i < numberOfValues; i++)
                {
                    values[i] = new ValutaValue { Time = today.AddHours(-i), Value = 8 * (i / 10) };
    
    
                }
                
                    var valuta = new Valuta()
                    {
                        Name = "GBP",
                        Values = new List<ValutaValue>(values)
                    };
    
    
                    TimeSpan timeUsed;
                    DateTime start = DateTime.Now;
                    store.Store(valuta);
                    store.Commit();
                    timeUsed = DateTime.Now - start;
    
    
                    Console.WriteLine("Time used inserting data: " + timeUsed.TotalMilliseconds);
                    Console.WriteLine("Press 'Enter' to continue....");
                    Console.ReadLine();
                
            }
        }
    
    
        public class Valuta
        {
            public string Id { get; set; }
            public string Name { get; set; }
            public IList<ValutaValue> Values { get; set; }
            
        }
    
    
        public struct ValutaValue
        {
            public DateTime Time { get; set; }
            public double Value { get; set; }
        }
    }
    
    Performance test of RavenDB saving and loading 240 000 valuta values instances in milliseconds:

    Saving operation:
    Time used inserting data: 12027,6
    Press 'Enter' to continue....

    Loading operation:
    Time used executing query: 14138,6
    Press 'Enter' to continue....

    Code:
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using Raven.Client.Document;
    using Raven.Client.Indexes;
    
    
    namespace ConsoleAppRavenTest
    {
        class Program
        {
            static void Main(string[] args)
            {
                var store = new DocumentStore { Url = "http://localhost:8080" };
                store.Initialize();
    
    
                Console.WriteLine("Press 'Enter' to read from database");
                ConsoleKeyInfo keyPressed = Console.ReadKey();
                if (keyPressed.Key == ConsoleKey.Enter) ReadFromDB(store);
                else InsertIntoDB(store);
    
    
                
    
    
            }
    
    
            private static void ReadFromDB(DocumentStore store)
            {
                /*store.DatabaseCommands.PutIndex("ValutaIndex", new IndexDefinitionBuilder<Valuta>
                {
                    Map = valutas => from valuta in valutas from v in valuta.Values select new { v.Time }
                });*/
    
    
                TimeSpan timeUsed;
                using (var session = store.OpenSession())
                {
    
    
                    //var all = session.Query<Valuta>().ToArray();
    
    
                    DateTime start = DateTime.Now;
                    var v = session.Query<Valuta>().SingleOrDefault(x => x.Name == "GBP");//Where(x => x.Name.Equals("GBP"))
                        //Advanced.LuceneQuery<Valuta>("ValutaIndex").Where("Name:GBP").WaitForNonStaleResults().Single();
                    timeUsed = DateTime.Now - start;
    
    
                    Console.WriteLine("Valuta: {0} | {1}", v.Id, v.Name);
                    foreach (var val in v.Values)
                    {
                        Console.WriteLine("Val: {0} at {1}", val.Value, val.Time);
                    }
                }
    
    
                Console.WriteLine("Time used executing query: "+timeUsed.TotalMilliseconds);
                Console.WriteLine("Press 'Enter' to continue....");
                Console.ReadLine();
            }
    
    
            private static void InsertIntoDB(DocumentStore store)
            {
                DateTime today = DateTime.Now.Date;
                int numberOfValues = 240000;
                ValutaValue[] values = new ValutaValue[numberOfValues];
                for (int i = 0; i < numberOfValues; i++)
                {
                    values[i] = new ValutaValue { Time = today.AddHours(-i), Value = 8 * (i / 10) };
    
    
                }
                using (var session = store.OpenSession())
                {
                    var valuta = new Valuta()
                    {
                        Name = "GBP",
                        Values = new List<ValutaValue>(values)
                    };
    
    
                    TimeSpan timeUsed;
                    DateTime start = DateTime.Now;
                    session.Store(valuta);
                    session.SaveChanges();
                    timeUsed = DateTime.Now - start;
    
    
                    Console.WriteLine("Time used inserting data: " + timeUsed.TotalMilliseconds);
                    Console.WriteLine("Press 'Enter' to continue....");
                    Console.ReadLine();
                }
            }
        }
    
    
        public class Valuta
        {
            public string Id { get; set; }
            public string Name { get; set; }
            public IList<ValutaValue> Values { get; set; }
            
        }
    
    
        public struct ValutaValue
        {
            public DateTime Time { get; set; }
            public double Value { get; set; }
        }
    
    
    }
    I'm not saying these databases are useless, I'm just saying that they are probably not meant for wast dataseries. As always, choose the appropriate tool for the job... ;-)

  6. #6
    Mark Pollack is offline Spring.NET Co-Lead Spring TeamSpring User
    Join Date
    Sep 2004
    Location
    New York, NY
    Posts
    1,683

    Default

    Hi,
    Interesting from the raw insert point of view, though is suspect what you might want to do with the data series might not be the best match for a relational db... Actually, just wanted to mention kdb from http://kx.com/ they seems to have a very high performance solution specifically for time series....
    Mark

  7. #7
    Join Date
    Oct 2006
    Location
    Bergen, Norway
    Posts
    365

    Default

    Hi Mark!

    Thanks, I'll be checking that out very soon

    Cheers,
    Steinar.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •