• Open Up Localhost

    In many cases it might be easier/better to just use LocalTunnel.

    In this case, I didn’t want the external URL changing all the time, and I already have a Linode server setup that I can use. Here are some notes on setting it up.

    Assumes a Linux server setup with SSH and Apache.

    Setup the proxy

    Enable the proxy modules on the server.

    a2enmod proxy
    a2enmod proxy_http
    

    Edit mods-available/proxy.conf to enable the reverse proxy for requests.

    ProxyRequests Off
    
    <Proxy *>
    Order deny,allow
    Allow from all
    </Proxy>
    

    Add a new virtual site to be the reverse proxy. Place a file like the following in the sites-available Apache directory. The localhost port doesn’t matter too much, just needs to not be in use.

    <VirtualHost *:80>
         ServerAdmin dustt
         ServerName virtual.red27.net
         SetEnv proxy-initial-not-pooled 1
         ProxyPass / http://localhost:8001
         ProxyPassReverse / http://localhost:8001
         ProxyPreserveHost On
    </VirtualHost>
    

    Enamble the virtual site and restart Apache.

    a2ensite virtual.red27.net
    

    On the client

    You will need to SSH into the server and reverse forward the packets back to the local server.

    ssh -nNT -R 8001:localhost:3000 user@virtual.red27.net
    

    This tunnel will need to be reset if the local server errors out. Removing the n argument may help notify you if something goes wrong.

  • Node.JS Modules

  • MongoDB 2012 Notes

    File and Data Structures

    • Key names are stored in the BSON doc, so make sure key names are short.
    • Data files are pre allocated, doubling in size each time.
    • Files are accessed using memory mapped files at the OS level.
    • fsync’d every 60 seconds.
    • Should use 64bit systems to support the memory mapped files.

    Journalling in 1.8 default in 2.0+

    • Write-ahead log
    • Ops written to journal before memory mapped regions
    • Journal flushed every 100ms or 100 MB written
    • db.getLastError({j:true}) to force a journal flush
    • /journal sub directory
    • 1 GB files, rotated ( only really need stuff that has not been fsync’d )
    • some slowdown on high throughput systems, can be symlink’d to other drives.
    • on by default in 64 systems.

    When to use

    • If Single node
    • Replica Set – at least 1 node
    • For large data sets.

    Fragmentation

    • files get fragmented over time if docs sizes change or deletes
    • collections that have a lot of resizes get padding factor to help reduce fragmentation.
    • need to improve free list
      • 2.0 reduced scanning to reasonable amount
      • 2.2 will change

    Compaction

    • 2.0+ compact command

      • only needs 2 GB extra space
      • off line operation (another good reason with replica sets.)
    • safemode: waits for a round trip from using getLastError, with this call you can specify how safe you want the data to be.

    • drop collection doesn’t free the data file, dropDatabase does. Sometimes it makes sense to create and drop databases.

    Index and Query Evaluation

    @mschireson

    • indexes are lists of values associated with documents
    • stored in a btree
    • required for geo queries and unique constraints
    • assending/descending really only matter on compound indexes.
    • null == null for unique indexes, you can drop duplicates on create.
    • create index is blocking, unless {background: true}, still should try to do off peak
    • when dropping an index, you need to use the same document when created.
    • the $where operator doesn’t use the indexes
    • Regexp’s starting with /^ will use an index
    • Indexes are used for updates and deletes
    • Compound indexes can be used to query for the first field and sort on the second
    • Only uses one index at a time
    • Limited index uses:
      • $ne uses the index, but doesn’t help performance much
      • $not
      • $where
      • $mod index only limits to numbers
      • Range queries only help some.

    GEO

    • created using “2d”
    • $near

      • sorted nearest to farthest
    • $within

    • $within{$polygon}}
    • can be in compound queries

    Sparse Indexes

    • only store values w/ the indexed field, results won’t have documents w/ null in that field.
    • can be sparse & unique

    Covering Indexes

    • contains all fields in the query and the results, no db lookup

    Limits and Trade offs

    • max of 64
    • can slow down inserts and updates
    • compound index can be more valuable and handle multiple queries
    • You can force an index or full scan
    • use sort if you really want sorted data
    • db.c.find(…).explain() => see whats going on.
    • db.setProfilingLevel() – record slow queries.
    • Indexes work best when they fit in RAM

    Replica Set

    • One is always the primary others are secondary.
    • Chosen by election
    • Automatic fail-over and recover
    • Reads can be from primary or secondary
    • Writes will always go to primary
    • Replica Sets are 2+ nodes, at least 3 is better
    • When a failed nodes come back, they recover by getting the missed updates, then join as a secondary node
    • Setup

      mongod —replSet

      cfg = { _id:, members: [{_id:0, host:‘’}] }

      use admin

      rs.initiate(cfg)

    • rs objects has replica set commands, needs to be issued on the current primary

    • rs.status()
    • Strong Consistency is only available when reading from primary
    • Reads on the secondary machines will be eventually consistent
    • Durability Options (set by driver)

      • fire and forget

        • won’t know about failures due to unique constraints, disk full, or anything else.
      • wait for error recommended

      • wait for journal sync
      • wait for fsync (slow)
      • wait for replication (really slow)
    • Can give nodes priorities, which will help ensure a specific machine is primary.

      • 0 priorities will never be primary
      • when a higher priority machine is back online, it will force an election.
    • Can have a slave delay

    • Tag replica sets with properties and can specify when waiting.
    • Arbiters Member

      • Don’t have data
      • vote in elections
      • used to break a tie
    • Hidden Member

      • not seen by the clients
    • Data is stored for replication in an oplog capped collection, all secondaries have an oplog too.

    Scaling

    • Vertical Scaling is limited
    • Horizontal scaling is cheaper, can scale wider then higher
    • Vertical can be a single point a failure, can be hard to backup/maintain.

    • Replica Sets are one type

      • Can scale reads, but now writes, eventual consistency is the biggest downside.
      • Replication can overwhelm the secondaries, reducing performance anyway
    • Why Shard?

      • Distribute the write load
      • Keep working set in RAM, by using multiple machines act like one big virtual machine
      • Consistent reads
      • Preserve functionality, by range based portioning most(all?) the query operators are available.
    • Sharding design goals

      • scale linearly
      • increase capacity with no downtime
      • transparent to the application / clients
      • low administration to add capacity
      • no joins or transactions
      • BigTable / PNUTS inspired read the PNUTS paper
    • Basics

      • Choose how you partition data
      • Convert from a single replica set to sharding with no downtime
      • Full feature set
      • Fully consistent by default
      • You pick a shard key, which is used to move ranges of data to a shard

    Architecture

    • Shard – each shard is it’s own replica set for automated fail over
    • Config Servers – store the meta data about where the partitions of data is, which shard

      • Not a replica set, writes to the config server is done with a transaction by the mongod / mognos
    • Mongos – uses the config servers to know what shard to use for the data/query

      • Client talks to the mongos servers
      • chunk = collection minkey, maxkey, shard
        • chunks are logical, not physical
        • chunk is 64MB, once you hit this point a new split happens, a new shard is created and data is moved.

    shard keys

    • they are immutable.
    • Choose a key
      • _id? is incremental this results in all writes going to one query
      • hash? is random, this partitions well, but now great for queries
      • user_id? kinda random, useful for lookups, all data for user_id X will be on one shard

        • However you can’t split on this if one user is a really heavy user.
      • user_id + md5(x)? this is the best option.

    Other notes

    • Want to add capacity way before it’s needed, at least before 70% operation capacity, this allows the data to migrate over time
    • Understand working set in RAM
    • Machine too small and admin overhead goes up
    • Machine too big and sharding doesn’t happen smoothly

    MMS – MongoDB Monitoring Service

    MMS

  • Include a Module Based on Rails Environment

    Not sure if this is the best way to do things. But I want to include modules into a class based on the rails environment.

    First I used the rails config store, Configurator.

    module MyApp
      class Application &lt; Rails::Application
        # ...
        config.provider = :NullProvider
      end
    end
    

    The above config can be overridden in the environment config file. NullProvider, would just be a noop implementation.

    Now, using this is a combo using send, accessing the config value, and getting the class from the symbol.

    class MyClass
      MyClass.__send__(:include, Kernel.const_get(::MyApp::Application.config.provider))
    end
    

    Put that where ever you’re include would normally go.

    Would love to know if this is something crazy or a good way to do things.

    References

  • Send Commands From VIM to Tmux Pane

    I’ve started using Tmux w/ VIM as my primary work flow. I installed the tslime.vim plugin, which will send highlighted commands to another pane, but I wanted to send ad-hoc commands, like :!. I added this function to the tslime.vim file, I’m sure there is a better way, but that’s what I did for now.

    function! To_Tmux()
      let b:text = input("tmux:", "", "custom,")
      call Send_to_Tmux(b:text . "\\r")
    endfunction
    
    cmap tt :call To_Tmux()<CR>
    

    This will allow me to type :tt and then any needed commmand, like rake. The tslime plugin will ask which pane number, then send the command to that pane from then on. The selected pane can be changed with <C-c>v.

    my forked tslime

  • ITerm2 TMux and Vim Setup

    Setup

    • Install MacVim
    • Install ITerm2
    • Install Tmux: brew install tmux
    • Install Tmux Patch for copy to OSX: tmux-MacOSX-pasteboard
    • Change shell to zsh chsh -s /bin/zsh && sudo chsh -s /bin/zsh username

    Tmux Config

    This sets some nice things for tmux. Of note, changing the default prefix to Ctrl+a and setting up the copy patch.

    /.tmux.conf

    set -g default-terminal "screen-256color"
    setw -g mode-mouse on
    set -g prefix C-a
    
    setw -g mode-keys vi
    
    bind h select-pane -L
    bind j select-pane -D
    bind k select-pane -U
    bind l select-pane -R
    
    bind-key -r C-h select-window -t :-
    bind-key -r C-l select-window -t :+
    
    # copy to osx
    set-option -g default-command "reattach-to-user-namespace -l zsh"
    bind ^y run-shell "reattach-to-user-namespace -l zsh -c 'tmux showb | pbcopy'"
    
    # quick pane cycling
    unbind ^a
    bind ^a select-pane -t :.+
    

    VIM

    Had to add

    syn on

    to my .vimrc to get syntax highlighting in console mode.

    Resources

  • Twitter and Facebook Popup Windows

    I often find my self needing to add Twitter and Facebook share buttons. Usually not using the default widget icons. To do that you need some functions to call window.open and some jQuery to tie the links to those functions. These functions also encode the passed parameters.

    var tweetWindow = function(url, text) {
      window.open( "http://twitter.com/share?url=" + 
        encodeURIComponent(url) + "&text=" + 
        encodeURIComponent(text) + "&count=none/", 
        "tweet", "height=300,width=550,resizable=1" ) 
    }
    
    var faceWindow = function(url, title) {
      window.open( "http://www.facebook.com/sharer.php?u=" + 
        encodeURIComponent(url) + "&t=" + 
        encodeURIComponent(title), 
        "facebook", "height=300,width=550,resizable=1" ) 
    }
    

    jQuery to enhance the link. This may or may not be used with the above.

    $(".twitter").click(function(e) {
        e.preventDefault();
        var href = $(e.target).attr('href');
        window.open(href, "tweet", "height=300,width=550,resizable=1") 
    });
    

    The link; using target _blank to work without JavaScript

    <a href="https://twitter.com/share?text=SOME%20TEXT&via=candland" 
        class="twitter" 
        target="_blank">Twitter Link</a>