Vincent Tang's Dev Blog

2016-10-13
Cheat-Sheet-ReferencePin

Vincent’s
List of sticky notes

Read More

2016-12-01
MS-Access-Preplan-Database

I figured out a preliminary database to build in MS-access. I didn’t have my MS-access database setup though properly and decided to just make screenshot annotations

I am not 100% sure how I will handle adding unique formula driven primary key. Like TRER-100 for TRER shortname brand + generic number identifier

Perhaps I should keep it simple and just combine it with the actual model#.

I need a way to generate unique values for model#s somehow PER manufacturer (autonumber)

Not sure how that works

Or calculated totals

I will also check out MySQL sample databases to see if it has more powerful intuitive tools (mySQL workbench)

I haven’t figured out how to add and update data and maintain consistency either

Probably through an excel workflow (reprocess garbage data) into access / mysql (highly refined data)

Resources

  • None

Vincent Tang

2016-11-30
SQL-Server-Optimal-Settings

Notes to myself on SQL-Server 2016 optimal settings:

Named pipes with the only setting (No TCP/IP or Shared memory) on SQL Express 2016 on my local machine reduced the time to download 540,000 entries in about a 1/10th the time and took approximately 5 minutes

The issue I had before was that something on the network stack was getting bottlenecked most likely. don’t know where though

On my MS-access filed linked to SQL-server 2016 it shows success

Resources

  • None

Vincent Tang

2016-11-29
SQL-Excel-Formula-Reference

This is a list of my most common SQL statements I found useful (either auto generated by access or made manually) as well as their applications in creating a datawarehouse

SQL

1
2
SELECT DISTINCT VendorShortName
FROM dbo_products;

For my work software, I ran this to create a list of vendor shortname unique primary Keys so that it would match my 500,000 row data import. A list of 100 vendors + were made. Saves a lot of time and consistency errors

1
2
SELECT A.Model
FROM A INNER JOIN dbo_products ON A.Model = dbo_products.Model

basically, “A” table has entries to put in. Inner join for values common between both tables, extract results. Similar to VLOOKUP in excel

I use this to extract model# I put in and grab out things like list prices

1
2
SELECT A.Model, dbo_products.ListPrice
FROM A INNER JOIN dbo_products ON A.Model = dbo_products.Model;

Adding dbo_products.ListPrice adds in ability to grab list price values. Can add as many columns of data here as needed. Just one example

Excel

1
=TRIM(IFERROR(MID(E146,FIND(",",E146,FIND("HP",E146)-13)+1,(FIND(",",E146,FIND("HP",E146))-FIND(",",E146,FIND("HP",E146)-13))-1),""))

This formula abstracts a messily compiled list of spec data and seperates them out into their own field. From there that data could be processed but I never ventured further.

The formula basically stops if an error is found. It looks at the midpoint of the text string, using a reference parameter “HP” for horsepower, and backtracks an arbitrary number of steps until it finds a comma. It goes back and forth between commas until HP is found

Its a stupidly complicated formula though and I think that a python script is probably much more efficient

Its fairly similar to a binary search algorithm though

Resources

  • None

Vincent Tang

2016-11-29
Building-A-Datawarehouse-With-MSSQL-and-MSaccess

2 weeks of work

Last 1-2 weeks (starting from Nov 2016?) I’ve been working on the MS-SQL database setup at work. I spent a lot of time learning about databases

I had to use a plethora of resources to understand and setup a datawarehouse. Classes I took in the mean time, mostly through lynda.com (since its organized)

  • Foundation of Networking: Networking basics
  • Foundations of Programming: Databases
  • Implementing a datawarehouse with Microsoft SQL 2012
  • Microsoft SQL Server 2016 Essential Training
  • MS Access 2016 essentials
  • Installing and Administrating Microsoft SQL Server 2016
  • Foundations of CS: Digital information
  • Foundations of CS: The Internet
  • Week 5 of harvard CS50

A lot of this information I had already known though, but I decided to relearn it anyhow just to understand. I picked these courses after auditing some 200+ courses and finding relevant information. Some courses I only finished partway or skimmed through relevant parts that I cared about. All notes I put into app.classeur.io, although some files got rather big (500 mB of ram to load a page), something I need to work on later

In addition to these resources, I posted questions on stackoverflow and reddit so I can understand these concepts more

  • https://dba.stackexchange.com/questions/156214/query-results-from-a-table-from-a-csv-file-list
  • https://dba.stackexchange.com/questions/156069/what-do-vertical-lines-refer-to-in-database-diagram-view

Mostly, I followed datawarehouse models found on http://www.databaseanswers.org/ from Barry Williams

The one I picked specifically to model are these:

Understanding the datamodel came from here:

A synposis of things I learned about databases from my notes:

  • ETL 1- extract transform load. Basically, extracting data from 1 database, transforming it through SQL queries or selecting specific areas, and loading into another application
  • ACID - atomic, consistency, isolation, durability. Describes interaction between 2 databases. ATOMICITY - if one part fails, everything fails. CONSISTENCY - triggers and cascades for database changes in effect. ISOLATION - Interaction between 2 databases is closed off to other applications during data transfer. DURABILITY - Prevent crashes and errors
  • CRUD - create, read, update, delete. Permissions given to users on databases
  • OLAP - online analytical processing - part of BI (excel pivot tables). Basically, takes data, tags them in different viewmodes (multidimensional database) to identify trends. Mostly used in large corporations, analysis, and BI http://searchdatamanagement.techtarget.com/definition/OLAP
  • OLTP - onine transacation processing: image,1. Essentially, processing data with queries using 3NF.
  • 1NF no repeating values or repeating groups. 2NF no nonkey values based on part of composite key. 3NF - no nonkey values based on other non-key values
  • EER vs ER diagrams. ER just shows basic workflow (Table relationships) but EER describes those relationships as well https://stackoverflow.com/questions/9906017/difference-between-er-diagram-and-eer-diagram
  • Referential integrity - every value in one table column must exist in another table column

OLAP I will just use for pricing optimization (NAW.org book), OLTP is for managing data between all systems (catalog, ecommerce), etc

Current setup:

Progress

So far this is what I’ve completed

  • Initial designs of RDBMS in MS-access
  • Integration with Data export software and setup of MS-SQL instance and software
  • Strategy for acquiring data from web competitors (web scraping)

Objective

My objective with the datawarehouse to improve the following processes at work:

  • Version control of data. Specs are all over the place and our excel catalog and work software costly time and money to fix and manage.
  • Ability to quickly query off results. By inputting a list of model# and abstracting data like bar codes, etc I can quickly load that data into a PoS software (and add inventory counts after)
  • OLTP, by downloading updated list prices from manufacturer I can adjust prices in ~November and ~March when companies update their prices by 5% to match into ecommerce and AdobeInDesign so no netloss in profits

Future

I intend to integrate as many data sources as possible, mostly on the free-end

There’s 3 types of data sources: - Internal data (work software import) - Creating data (pricing and markups) - External data (web scraping)

I want to focus on managing data and abstracting external data in the future. Mostly, data for category landing pages (e.g. filter by manufacturer, voltage, etc) since the work software does not organize this easily. I outsourced an advanced excel formula but even then =TRIM(IFERROR(MID(E146,FIND(",",E146,FIND("HP",E146)-13)+1,(FIND(",",E146,FIND("HP",E146))-FIND(",",E146,FIND("HP",E146)-13))-1),"")) that one isn’t good enough

Also, the category layouts, I don’t know how I will implement a relational-database layout for this. Each category has different… filters for searching and data in those filters. It doesn’t change over time since its fixed.

So my goals are:

  • Frameworks for scraping data
  • Building relational database warehouse further

Resources

  • None

Vincent Tang

2016-11-19
Datawarehouse-Plan-Work

Generally dataflow for work, RDBMS setup usingn MSaccess/excel for small manual data input, excel for processing garbage data into nicely refined data, and Microsoft SQL 2016 locally to manage queries for data

Some information not entirely sure of, such as the relationships between MSaccess vs. Microsoft SQL 2016 and general querying structures

Will figure it out as I need too

Layout of RDBMS and relationships of data, I’ll keep it mostly done via MSaccess since I’m the only user using it. Most queries controlled here if exporting to excel.

MS-SQL is mostly to run ETL for databases to get updated list prices, and if needed manage ODBC connection to adobe indesign/ 65 bit software

I do not intend to have any relationships between Excel to MySQL most likely to keep it simple (KISS)

Resources

  • None

Vincent Tang

2016-11-17
ClasseurIO-And-Imagus-Settings

I have my setup down pretty well when it comes to making notes. The method is basically

  1. Greenshot editor to make image diagrams
  2. ShareX to make gif image notes for things like adobe premierePro notes
  3. Imgur to host images
  4. App.Classeur.IO to add contextual notes to those images/gifs and organize
  5. Imagus to preview images

The problem I had through app.classeur.io wasn’t yielding the results I wanted. I wrote a post on it

https://softwarerecs.stackexchange.com/questions/37580/copy-image-url-address-open-new-tab-chrome-plugin

Essentially what was happening was I couldn’t seamless integrate viewing my notes after I wrote them without constantly opening new tabs. Because the diagrams were too small and hard to read and I had to zoom up to see it

I tried editing imagus settings but the only shortcut I found useful was the “O” key.

My “sieve” settings as well are here

This didn’t help out because sometimes I’d press the “O” key and wasn’t enabling the shortcut correctly

I ended up for now, just adjusting the binderwidth here:

so that it would be easier for me to see notes in “preview mode”

Its not a perfect solution though.

another solution I found out I could just open a new tab instead. Didn’t think about that

The solution just ended up being “open image in new tab”. Can’t believe I didn’t recognize that basic command

Resources

  • None

Vincent Tang

2016-11-09
CS50-Problem-Set-4-Solutions

PSET 4 solutions for cs50 class

Read More

2016-11-04
Learn-Apply-Reflect

Older notes

This is my current notetaking methodology for long term retention of notes, and how my brain is processing information in real time

The basic principle is such:

  1. Learn - Structured
  2. Apply - Unstructured
  3. Reflect - Structured

This is how my brain understands and processes information, broken down into 3 categories: 1. How the brain Works. 2. Philosophy of Learning. 3. Tools to enhance the process

1. Learn

Philosophy

Learning is like growing knowledge nodes into trees. Basically, I learn from people that are smarter than me, in different fields of applications. Usually these people will be experts in the industry. Say Stephen Hawking or Tom Scott for instance. I would learn it through a platform that inherently organizes the information, say lynda.com. Its inherently structured normally unless whoever is organizing the information does it poorly, in that event LEARN - > APPLY are cycled back and forth.

In many ways this is a top down learning approach

Brain

The brain works by associating knowledge from others to yourself. Writing down notes helps in that process. http://knowyourmeme.com/photos/1079173-i-made-this.

Tools

Tools include youtube, lynda.com, MooC or any authoritative online resource. You can play it back rewind video at your leisure to understand the concepts being taught, depending on how difficult. Or even notes from textbooks too. Or Ama forum posts from smart people in that field. I take my notes via app.classeur.IO and shareX in a scalable method. See post http://www.tangycode.com/My-Notetaking-Method-Reddit-Post/ more information

2. Apply

Philosophy

This is inherently unstructured. Its like googling an issue to a computer problem that only you seem to have. You have to basically write a paper from scratch and source all the information you had to get in order to solve a difficult problem. Like building a cloned version of minecraft. There might not be a lot of documentnation on it, so you have to ask lots of questions on stackoverflow and create project log notes of your thought process unless (KXCD reference programmer interrupted). http://imgur.com/gallery/3uyRWGJ

In many ways this is a bottoms up learning approach

Brain

The brain works by forming synapses, or associations of knowledge from different areas. People who have many synapsues / many hobbies tell better jokes in general and make better analogies as a whole.

Tools

Tools include asking questions on stackoverflow, capturing batshit crazy ideas to the moon, creating a fast sprinting log for notes that might be super long or really short - depending on how many steps it took to solve a problem, prose.IO +shareX+jekyll blog for capturing sprint notes

3. Reflect

Philosophy

This goes into understanding that the brain can only retain so much information at a time. There’s been many studies on this although I didn’t reference them. Basically, there’s something called a forgetting curve, which is inversely exponentially to time. Things you learn are fresh in your mind - things that you haven’t thought about or learned are not, and take sometime to refresh those notes

This is inherently structured, because you understand what the information and are trying to convey it to someone who doesn’t (r/ELI5 on reddit)

Brain

The brain works as node tend to degenerate over time if you don’t form associations with other notes, and reinforce those synapsuses

Tools

Tools include creating youtube videos using AdobePremierePro and AfterEffects. Other ways is to write a blog(personal, or not personal) to relay information to others. OR answering questions on stackoverflow, reddit, quora, etc. Or teaching and tutoring someone in real life. For me, its making diagrams and writing personal dev blog notes on this blog. Everyone has a different preference


Summary

This is how my current understanding of note methodologies work from my own perspective. Sometimes I have these urge thoughts that I HAVE TO WRITE down, whether its just randomly occuring to me, r/luciddreams style. For me its important because I firmly believe in capturing those breakthrough ideas and thoughts reinforces the brains ability to retain and process information, thereby allowing you to be smarter than you were before

No studies really have confirmed any of this to be true to my knowledge, but from my own ancedotal experiences and personal studies it seems to be this way. That’s what counts

Resources

Vincent Tang

2016-11-03
Sleep-Issue-Log

I tried everything I read up about online. I’ve had sleep issues most of my life. I don’t know how many hours of research I’ve put into this

I find the following combinations works well for falling asleep:

  • Reading reddit at night using google app “twilight” works really well for falling asleep
  • Nasal bandage breathing aids. I always have congested nose problems
  • 1.5 hours before sleeping, no working out or eating
  • Doing intensive learning or busywork 1-2 hours before sleeping helps
  • Temp around 75*F is ideal
  • Signs of drooling are generally a good sign of sleep
  • 11-11-16. Nose vents are much more effective than nasalbandage breathing aids here

I find the following makes it difficult to fall asleep

  • ********* it out before sleeping doesn’t help me
  • Watching TV shows on my phone
  • Reading fiction books in general only makes me more interested
  • Melantonin doesn’t really work for me
  • Reading a boring book like the history of LEAN (paperback) works, but isn’t an effective use of time
  • Showering before sleeping doesn’t help, having natural oils on skin promotes sleeping for me
  • Getting the **** kicked out of me during sparring practice makes it difficult to sleep sometimes

I find mixed results and or not enough information on the following:

  • Lucid dreaming and or dream states usually are in ?deep sleep states?
  • Waking up on 2 alarm timer resets, 10 min intervals, in the event sleep was bad and need a 20 min nap
  • Lasko low blowing fan helps sleep by disappating heat more, however it dries out my throat though potentially making me sick
  • Left side or right side sleeping. Supposedly left side is supposed to be better for you, but in all honesty I find muscle recovery faster in neutral position.

Other things I found out

  • 11-11-16. I have trouble breathing because my dad grew up chain smoking, and I habitually stop breathing every now and then and this seems to affect me in sleep (nose vents alleviated problem)
  • I may have nasal poylps because I cannot breathe on one side of my nose my entire life.
  • I think I may have medical issues b/c sometimes I feel abnormally tired. I am not sure

Resources

  • None

Vincent Tang

2016-11-03
GDB-Reference-Sheet

This is a cheatsheet of how to utilize GDB for cs50 and other applications through the command line

NOTES:

  • GDB stands for global debugger
  • It works on Linux OS, sometimes used with VI or VIM
  • GDB only works on executable, not source files like .c
  • Runs on the terminal, cmd prompt, bash, etc

Running GDB is done by the following, navigate to the file first using cd

1
$ GDB ./copy

At this point, can specify any argv[1] or argv[2] things to add on (inputs)

1
$

./copy, where copy is the executable file

1
break main

add a break in the main function

1
$ r

R stands for run, runs the program

1
$ quit

CTRL+C does not work to exit out of GDB


Debugger vs GDB

Debugger through CS50 is more-so on the IDE side

  • Works on source file
  • Compiles the program in real time

Resources

  • http://cs50.stackexchange.com/questions/21681/how-do-you-pass-arguments-into-gdb-argv1-argv2/21682#21682, my question on stackoverflow great answer
  • https://www.reddit.com/r/cs50/comments/44x3dy/program_command_line_args_using_ide_gdb/

Vincent Tang