{"id":559,"date":"2020-07-28T13:00:00","date_gmt":"2020-07-28T17:00:00","guid":{"rendered":"https:\/\/www.pwvconsultants.com\/blog\/?p=559"},"modified":"2020-07-27T15:46:36","modified_gmt":"2020-07-27T19:46:36","slug":"log-stores-are-specialized-databases","status":"publish","type":"post","link":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/","title":{"rendered":"Log Stores Are Specialized Databases"},"content":{"rendered":"\n<p>This installment of our <a href=\"https:\/\/www.pwvconsultants.com\/blog\/datastores-an-overview-of-common-types\/\" target=\"_blank\" aria-label=\"undefined (opens in a new tab)\" rel=\"noreferrer noopener\">series on datastores<\/a> addresses one of the most-used databases in business today: The log store. Log stores are very specialized and have a very specific purpose. Using it for anything outside that purpose is going to make the store inefficient and cause you headaches down the road.<\/p>\n\n\n\n<p>Log stores are meant to be storage for quick inserting and streaming in data. It\u2019s generally time series-ed and ingested in a way where one of two things are happening. Either the data is being consumed by another system and then shoved somewhere to log it, which means it\u2019s not live or  it\u2019s streaming in, either the system is reading from something that is streaming in or is being posted to. Either way there are files that are filling with logs on a system which get pulled into a SaaS system or other outside log store. The idea is that as they\u2019re getting pulled in, there are regular expressions parsing the data and giving it context with the expectation that a certain log prints or aligns its data in a certain format. You will know where the date is going to be, where a specific type of data is in a printed error statement, and you can surface the data and sort it from what would otherwise be a never ending log file.<\/p>\n\n\n\n<p>The reason why this is important is it is meant to take simple data that&#8217;s usually line delimited and put it somewhere it can be filtered to get eyes on it. Log stores are not meant to do complicated queries and cross-sections, they aren\u2019t meant to delete data or certainly not handle a data life cycle. They are literally meant for logs, to take a quick snapshot of what is happening so that someone can view it and know what the data is doing. If you want to look at the data from two days ago, the log store is the way to do it. You generally do not look at log store data in real-time, unless you put  programmatic spies on the data so that it\u2019s being compared to criteria as it comes in. You can then create alerts, alarm points, or have automated actions take place based on specific matches. <\/p>\n\n\n\n<p>The other major advantage of log stores is centralization and aggregation. Often you will take data from multiple systems that have logs in different places. Essentially, you\u2019d have to go into each system to look at them and start pulling down files. But with a log store you can see a snapshot of time across multiple log sources to determine where something started, what systems are affected and figure out what problems might be happening. It centralizes what you are doing.<\/p>\n\n\n\n<p>Log stores can handle massive amounts of data and can filter a lot of data very rapidly (remember you&#8217;re meant to be looking at day from two days ago &#8211; so rapid may be 15 minutes to get a result), which is why people tend to misuse them. They want to use a log store or an event log for relational purposes, but a log store simply doesn\u2019t work that way. You don\u2019t want to look at patterns in a log store because all you can do is filter for a specific event, there\u2019s no other relationship that can be actively derived. Spying on live data and reacting to it is one thing, but it doesn\u2019t mean it can do other things efficiently. The more data you put into it, the longer it takes for queries to return.<\/p>\n\n\n\n<p>Log stores are limited in their functionality, but they are very good and very efficient at what they are designed to do. Every business should have a log store of some kind to track events and help locate when and where a problem occurred. Just don\u2019t try to do more with them because it\u2019s going to get hung up and not work properly.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.<\/p>\n","protected":false},"author":1,"featured_media":583,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[961,958],"tags":[1389,35,1388,925,1272,1387,1279],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v22.2 (Yoast SEO v22.2) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Log Stores Are Specialized Databases - PWV Consultants<\/title>\n<meta name=\"description\" content=\"Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Log Stores Are Specialized Databases\" \/>\n<meta property=\"og:description\" content=\"Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\" \/>\n<meta property=\"og:site_name\" content=\"PWV Consultants\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/PWV-Consultants-110444033947964\" \/>\n<meta property=\"article:published_time\" content=\"2020-07-28T17:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-07-27T19:46:36+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1275\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Pieter VanIperen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PWV_Consultants\" \/>\n<meta name=\"twitter:site\" content=\"@PWV_Consultants\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pieter VanIperen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\"},\"author\":{\"name\":\"Pieter VanIperen\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4\"},\"headline\":\"Log Stores Are Specialized Databases\",\"datePublished\":\"2020-07-28T17:00:00+00:00\",\"dateModified\":\"2020-07-27T19:46:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\"},\"wordCount\":675,\"publisher\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg\",\"keywords\":[\"aggregation\",\"Automation\",\"centralization\",\"data\",\"Datastore\",\"event log\",\"Log Store\"],\"articleSection\":[\"Data Stores\",\"Information Technology\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\",\"name\":\"Log Stores Are Specialized Databases - PWV Consultants\",\"isPartOf\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg\",\"datePublished\":\"2020-07-28T17:00:00+00:00\",\"dateModified\":\"2020-07-27T19:46:36+00:00\",\"description\":\"Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg\",\"contentUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg\",\"width\":1920,\"height\":1275},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pwvconsultants.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Log Stores Are Specialized Databases\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#website\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/\",\"name\":\"PWV Consultants\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pwvconsultants.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#organization\",\"name\":\"PWV Consultants\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png\",\"contentUrl\":\"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png\",\"width\":98,\"height\":84,\"caption\":\"PWV Consultants\"},\"image\":{\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/PWV-Consultants-110444033947964\",\"https:\/\/twitter.com\/PWV_Consultants\",\"https:\/\/www.linkedin.com\/company\/pwv-consultants\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4\",\"name\":\"Pieter VanIperen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g\",\"caption\":\"Pieter VanIperen\"},\"description\":\"PWV Consultants is a boutique group of industry leaders and influencers from the digital tech, security and design industries that acts as trusted technical partners for many Fortune 500 companies, high-visibility startups, universities, defense agencies, and NGOs. Founded by 20-year software engineering veterans, who have founded or co-founder several companies. PWV experts act as a trusted advisors and mentors to numerous early stage startups, and have held the titles of software and software security executive, consultant and professor. PWV's expert consulting and advisory work spans several high impact industries in finance, media, medical tech, and defense contracting. PWV's founding experts also authored the highly influential precursor HAZL (jADE) programming language.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/pwv-consultants\"]}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Log Stores Are Specialized Databases - PWV Consultants","description":"Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/","og_locale":"en_US","og_type":"article","og_title":"Log Stores Are Specialized Databases","og_description":"Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.","og_url":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/","og_site_name":"PWV Consultants","article_publisher":"https:\/\/www.facebook.com\/PWV-Consultants-110444033947964","article_published_time":"2020-07-28T17:00:00+00:00","article_modified_time":"2020-07-27T19:46:36+00:00","og_image":[{"width":1920,"height":1275,"url":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg","type":"image\/jpeg"}],"author":"Pieter VanIperen","twitter_card":"summary_large_image","twitter_creator":"@PWV_Consultants","twitter_site":"@PWV_Consultants","twitter_misc":{"Written by":"Pieter VanIperen","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#article","isPartOf":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/"},"author":{"name":"Pieter VanIperen","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4"},"headline":"Log Stores Are Specialized Databases","datePublished":"2020-07-28T17:00:00+00:00","dateModified":"2020-07-27T19:46:36+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/"},"wordCount":675,"publisher":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage"},"thumbnailUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg","keywords":["aggregation","Automation","centralization","data","Datastore","event log","Log Store"],"articleSection":["Data Stores","Information Technology"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/","url":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/","name":"Log Stores Are Specialized Databases - PWV Consultants","isPartOf":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage"},"image":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage"},"thumbnailUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg","datePublished":"2020-07-28T17:00:00+00:00","dateModified":"2020-07-27T19:46:36+00:00","description":"Log stores are meant for one thing: To log events and specific information. Using a log store for anything else renders it inefficient and slow.","breadcrumb":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#primaryimage","url":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg","contentUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/07\/Time-Series-and-Flat-File.jpg","width":1920,"height":1275},{"@type":"BreadcrumbList","@id":"https:\/\/www.pwvconsultants.com\/blog\/log-stores-are-specialized-databases\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pwvconsultants.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Log Stores Are Specialized Databases"}]},{"@type":"WebSite","@id":"https:\/\/www.pwvconsultants.com\/blog\/#website","url":"https:\/\/www.pwvconsultants.com\/blog\/","name":"PWV Consultants","description":"","publisher":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pwvconsultants.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.pwvconsultants.com\/blog\/#organization","name":"PWV Consultants","url":"https:\/\/www.pwvconsultants.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png","contentUrl":"https:\/\/www.pwvconsultants.com\/blog\/wp-content\/uploads\/2020\/04\/logo-alternate-e1585773530392.png","width":98,"height":84,"caption":"PWV Consultants"},"image":{"@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/PWV-Consultants-110444033947964","https:\/\/twitter.com\/PWV_Consultants","https:\/\/www.linkedin.com\/company\/pwv-consultants"]},{"@type":"Person","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/c15d5d40126a8ad906cb3067de95f8d4","name":"Pieter VanIperen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pwvconsultants.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8b294918257a810803e2befc9a71b7bc?s=96&d=mm&r=g","caption":"Pieter VanIperen"},"description":"PWV Consultants is a boutique group of industry leaders and influencers from the digital tech, security and design industries that acts as trusted technical partners for many Fortune 500 companies, high-visibility startups, universities, defense agencies, and NGOs. Founded by 20-year software engineering veterans, who have founded or co-founder several companies. PWV experts act as a trusted advisors and mentors to numerous early stage startups, and have held the titles of software and software security executive, consultant and professor. PWV's expert consulting and advisory work spans several high impact industries in finance, media, medical tech, and defense contracting. PWV's founding experts also authored the highly influential precursor HAZL (jADE) programming language.","sameAs":["https:\/\/www.linkedin.com\/company\/pwv-consultants"]}]}},"_links":{"self":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts\/559"}],"collection":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/comments?post=559"}],"version-history":[{"count":5,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts\/559\/revisions"}],"predecessor-version":[{"id":584,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/posts\/559\/revisions\/584"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/media\/583"}],"wp:attachment":[{"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/media?parent=559"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/categories?post=559"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pwvconsultants.com\/blog\/wp-json\/wp\/v2\/tags?post=559"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}