techcrunch.py (88479ef) - techcrunch.git

88479ef0c1b447f50560fe88af76f50ac82324cd

Catch up to production agai... David Blume authored 6 years ago	`1) #!/usr/bin/env python`
Original 2010-09-03 version David Blume authored 6 years ago	`2) #`
Moved SMTP credentials to s... David Blume authored 6 years ago	`3) # Testing without affecting the yaml file and saving the updated one aside: 4) # cp techcrunch.yaml techcrunch.yaml_back; ./techcrunch.py; \ 5) # cp techcrunch.yaml techcrunch_tmp.yaml; cp techcrunch.yaml_back techcrunch.yaml`
Original 2010-09-03 version David Blume authored 6 years ago	`6) import feedparser 7) import sys`
Upgrade yaml module dblume authored 5 months ago	`8) if sys.path[0] != '': 9) sys.path.insert(0, '') # DXB Only needed for cronjobs to find yaml 10) import yaml`
Original 2010-09-03 version David Blume authored 6 years ago	`11) import os 12) import time 13) import codecs 14) import traceback 15) import calendar 16) import pickle 17) import exceptions 18) import urllib 19) import urllib2 20) import httplib 21) import shutil 22) import smtplib 23) import analysis`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`24) import json`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`25) import xml`
2011-02-04: Update to the c... David Blume authored 6 years ago	`26) import operator`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`27) import cgi`
Google terminated image cha... David Blume authored 5 years ago	`28) import cStringIO`
Moved SMTP credentials to s... David Blume authored 6 years ago	`29) import smtp_creds # Your own credentials, used in send_email()`
Original 2010-09-03 version David Blume authored 6 years ago	`30) 31) debug = True 32) any_entry_added = False`
Google terminated image cha... David Blume authored 5 years ago	`33) tags_to_post = {'apple', 'google', 'roku'}`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`34) authors_to_post = ['michael arrington',] 35)`
Upgrade yaml module dblume authored 5 months ago	`36) # 2022-02-24: Seeing too many posts with just mean + sigma, adjust with factor 37) threshold_sigma_factor = 1.5 38)`
Catch up to production agai... David Blume authored 6 years ago	`39) # TODO 2018-01-18: Maybe combine fb_likes with bf_shares or something... 40) rhs_metric = 'fb_likes'`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`41) rhs_metric_times = 'comment_times'`
Original 2010-09-03 version David Blume authored 6 years ago	`42) 43) localdir = '' 44)`
Google terminated image cha... David Blume authored 5 years ago	`45) html_head = """<!DOCTYPE html> 46) <html><head>`
Original 2010-09-03 version David Blume authored 6 years ago	`47) <title>TechCrunch Feed Filter</title> 48) <!-- <link rel="alternate" type="application/rss+xml" title="RSS feed" href="http://techcrunch.dlma.com/rss_feed.xml" /> --> 49) <link rel="alternate" type="application/rss+xml" title="RSS feed" href="http://feeds.feedburner.com/TrendingAtTechcrunch" /> 50) <style type="text/css"> 51) body { font-family: "Arial", san-serif; }`
Fix comment, and update a URL. David Blume authored 5 years ago	`52) .author { font-size: smaller; color:gray; } 53) .date { font-size: smaller; color:gray; }`
Original 2010-09-03 version David Blume authored 6 years ago	54) .h3 { font-size: larger; } 55) a { text-decoration: none; } 56) /* table { border: none; border-collapse:collapse; font-size: large } */ 57) table { border-collapse: collapse; } 58) table.legend { border:1px solid LightSlateGray; font-size: medium; border-collapse:separated; } 59) table.legend th { border: 1px solid LightSlateGray; background-color: #E0E0E0; } 60) table.legend td { border: 1px solid LightSlateGray; } 61) tr.even { background:#%s; padding: 2em; } 62) tr.odd { background:#%s; padding-bottom: 2em; }
Little cleanup, div to rese... David Blume authored 5 years ago	`63) td div { height: 68px; }`
Original 2010-09-03 version David Blume authored 6 years ago	`64) </style>`
Google terminated image cha... David Blume authored 5 years ago	`65) <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script> 66) <script type="text/javascript"> 67) google.charts.load('current', {'packages':['corechart']}); 68) google.charts.setOnLoadCallback(drawChart); 69) function drawChart() { 70) var options = {`
Fix comment, and update a URL. David Blume authored 5 years ago	`71) width:%d,`
Google terminated image cha... David Blume authored 5 years ago	`72) height:68, 73) pointSize:0.1, 74) dataOpacity:1.0, 75) series: { 0: {targetAxisIndex:0}, 1: {targetAxisIndex:1, color:'limegreen'} }, 76) vAxis: { gridlines: {count: 0}, maxValue: 1 }, 77) hAxis: { gridlines: {count: 0}, ticks: [] }, 78) vAxes: { 0: {textStyle: {fontSize: 11, color: 'blue'} }, 1: {viewWindowMode: 'maximized', baselineColor: '#A0D0A0', textStyle: {fontSize: 11, color: 'limegreen'} } }, 79) }; 80) %s 81) } 82) </script> 83) </head> 84) <body>`
Original 2010-09-03 version David Blume authored 6 years ago	`85) <div align='center'><h3>TechCrunch Feed Filter</h3></div>`
Add feed icon and link to t... David Blume authored 6 years ago	`86) This page shows what analysis is done to filter the noise away from the Techcrunch feed into 87) <a href="http://feeds.feedburner.com/TrendingAtTechcrunch"> a more concise feed <img src="feed.png" alt="feed" height="14" width="14"></a>.`
Google terminated image cha... David Blume authored 5 years ago	`88) <a href="https://david.dlma.com/blog/my-techcrunch-feed-filter">Learn more about the Feed Filter</a>.<br /><br />`
Original 2010-09-03 version David Blume authored 6 years ago	`89) """ 90) 91) html_footer = """ 92) </table> 93) </div><br /> 94) <div align='center'>Thanks to <a href="http://www.feedparser.org/">The Universal Feed Parser module</a>,`
Add feed icon and link to t... David Blume authored 6 years ago	`95) <a href="http://pyyaml.org/">PyYAML</a> and <a href="http://code.google.com/apis/chart/">Google Charts</a>.<br />`
Google terminated image cha... David Blume authored 5 years ago	96) <a href="http://git.dlma.com/techcrunch.git/">source</a> • <a href="techcrunch.yaml">raw data</a> • <a href="stats.txt">status</a><br />© 2011 <a href="https://david.dlma.com">David Blume</a></div><br /> 97) </body> 98) </html> 99) """ 100) 101) chart_data_header = """ var data = google.visualization.arrayToDataTable([ 102) ['', 'Comments', 'Shares', {'type': 'string', 'role': 'style'}], 103) """ 104) chart_data_middle = """ ]); 105) var chart = new google.visualization.LineChart(document.getElementById('chart%d')); 106) options.backgroundColor = '#%s';
Original 2010-09-03 version David Blume authored 6 years ago	`107) """ 108)`
Fix comment, and update a URL. David Blume authored 5 years ago	`109) img_width = 240`
Google terminated image cha... David Blume authored 5 years ago	`110) img_height = 68`
Original 2010-09-03 version David Blume authored 6 years ago	`111) 112) series_1_color = "0000FF" 113) series_2_color = "00AA00" 114) threshold_color = "FF8C00"`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`115) tag_color = "F01000"`
Original 2010-09-03 version David Blume authored 6 years ago	`116) 117) even_background = "F8F8F8" 118) odd_background = "E8E8E8" 119)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`120) even_watermark = "E0E0FF" 121) odd_watermark = "D0D0F0" 122) 123)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`124) def asciiize(s):`
Original 2010-09-03 version David Blume authored 6 years ago	`125) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`126) return s.encode('ascii')`
Better conformance to PEP-8... David Blume authored 6 years ago	`127) except UnicodeEncodeError:`
Original 2010-09-03 version David Blume authored 6 years ago	`128) return s`
Better conformance to PEP-8... David Blume authored 6 years ago	`129) except exceptions.AttributeError:`
Original 2010-09-03 version David Blume authored 6 years ago	`130) return s 131)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`132)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`133) def send_email(subject, message, toaddrs, 134) fromaddr='"%s" <%s>' % (os.path.basename(__file__), smtp_creds.user)): 135) """ Sends Email """ 136) smtp = smtplib.SMTP(smtp_creds.server, port=smtp_creds.port) 137) smtp.login(smtp_creds.user, smtp_creds.passw)`
Better conformance to PEP-8... David Blume authored 6 years ago	`138) smtp.sendmail(fromaddr, 139) toaddrs,`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`140) "Content-Type: text/plain; charset=\"us-ascii\"\r\nFrom: %s\r\nTo: %s\r\nSubject: %s\r\n%s" % \ 141) (fromaddr, ", ".join(toaddrs), subject, message))`
Original 2010-09-03 version David Blume authored 6 years ago	`142) smtp.quit() 143)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`144)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`145) def index_id(a_list, op, elem):`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`146) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`147) return (index for index, item in enumerate(a_list) if op(item, elem)).next()`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`148) except: 149) return -1 150)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`151)`
Google terminated image cha... David Blume authored 5 years ago	`152) def write_chart_data(time_posted, lhs_times, lhs_values, rhs_times, 153) rhs_values, threshold_value, image_index, tag_hit, chart_io):`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`154) # lhs_times, lhs_values = zip(comments) 155) # rhs_times, rhs_values = zip(rhs)`
Google terminated image cha... David Blume authored 5 years ago	`156) is_odd_row = image_index % 2`
Original 2010-09-03 version David Blume authored 6 years ago	`157)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`158) if not len(lhs_times): 159) lhs_times = [time_posted,] 160) if not len(lhs_values): 161) lhs_values = [0,] 162) if not len(rhs_times): 163) rhs_times = [time_posted,] 164) if not len(rhs_values): 165) rhs_values = [0,] 166) 167) lhs_times = [(i - time_posted) / 1800 for i in lhs_times] 168) rhs_times = [(i - time_posted) / 1800 for i in rhs_times] 169)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`170) met_threshold_pt = -1 171) if threshold_value != -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`172) met_threshold_pt = index_id(rhs_values, operator.ge, threshold_value)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`173) if met_threshold_pt == -1 or tag_hit: 174) # This can happen if threshold_value was set to a number 175) # because the author or a tag was matched, but the article 176) # was unpopular. We choose to put a marker at point index 0. 177) met_threshold_pt = 0 178) 179) if is_odd_row != 0: 180) bg_color = even_background 181) else: 182) bg_color = odd_background`
Google terminated image cha... David Blume authored 5 years ago	`183) 184) chart_io.write(chart_data_header) 185) for i in range(8): 186) if i == met_threshold_pt: 187) if tag_hit: 188) style = "'point { size: 5; fill-color: #FF0000; shape-type: diamond}'" 189) else: 190) style = "'point { size: 5; fill-color: #FF8C00; }'" 191) else: 192) style = "null" 193) if i < len(lhs_values): 194) lhs_value = str(lhs_values[i]) 195) else: 196) lhs_value = "null" 197) if i < len(rhs_values): 198) rhs_value = str(rhs_values[i])`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`199) else:`
Google terminated image cha... David Blume authored 5 years ago	`200) rhs_value = "null" 201) chart_io.write(" [%d, %s, %s, %s],\n" % (i, lhs_value, rhs_value, style)) 202) chart_io.write(chart_data_middle % (image_index, bg_color)) 203) if met_threshold_pt == -1 and not tag_hit: 204) chart_io.write(" delete options.vAxes[1].baseline;\n") 205) else: 206) chart_io.write(" options.vAxes[1].baseline = %d;\n" % (threshold_value,)) 207) chart_io.write(" chart.draw(data, options);\n\n")`
Original 2010-09-03 version David Blume authored 6 years ago	`208)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`209)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`210) def process_feed(yaml_items):`
Moved SMTP credentials to s... David Blume authored 6 years ago	`211) """Retrieve the url and process it.`
Original 2010-09-03 version David Blume authored 6 years ago	`212) feed_info (in, out) A tuple that describes an individual feed, like its name and etag. 213) """`
Upgrade yaml module dblume authored 5 months ago	`214) #feed = feedparser.parse('https://techcrunch.com/feed/') 215) feed = feedparser.parse('https://pi.dlma.com/techcrunch_feed.php') # DXB temporary until removed from denylist`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`216) if hasattr(feed, 'status'):`
Original 2010-09-03 version David Blume authored 6 years ago	217) if feed.status == 304: 218) pass 219) else: 220) if feed.status != 200 and feed.status != 307 and feed.status != 301 and feed.status != 302: 221) if feed.status == 503: 222) print "the feed is temporarily unavailable." 223) elif feed.status == 400: 224) print "the feed says we made a bad request." 225) elif feed.status == 502: 226) print "the feed reported a bad gateway error." 227) elif feed.status == 404: 228) print "the feed says the page was not found." 229) elif feed.status == 500: 230) print "the feed had an internal server error." 231) elif feed.status == 403: 232) print "Access to the feed was forbidden." 233) else: 234) print "the feed returned feed.status %d." % ( feed.status, ) 235) else: 236) # Save off this
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`237) if hasattr(feed, 'bozo_exception') and isinstance(feed.bozo_exception, xml.sax._exceptions.SAXParseException): 238) print "Didn't pickle TechCrunch feed because it had a bozo_exception: %s" % (str(feed.bozo_exception))`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`239) else: 240) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`241) with open(os.path.join(localdir, 'techcrunch_feed.pickle'), 'wb') as f: 242) pickle.dump(feed, f)`
Catch up to production agai... David Blume authored 6 years ago	`243) except(pickle.PicklingError, exceptions.TypeError) as e:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`244) print "An error occurred while pickling the feed: %s." % \`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`245) (# str(e.__class__), 246) str(e)) 247) traceback.print_exc(3, file=sys.stdout)`
Original 2010-09-03 version David Blume authored 6 years ago	`248)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`249) for i in reversed(feed.entries): 250) process_item(i, yaml_items)`
Original 2010-09-03 version David Blume authored 6 years ago	`251) 252) # If we have more than 200 items, remove the old ones.`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`253) while len(yaml_items) > 200:`
Original 2010-09-03 version David Blume authored 6 years ago	`254) yaml_items.pop() 255) 256) for i in yaml_items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`257) # i['title'] = asciiize(i['title']) 258) # i['tags'] = map(asciiize, i['tags']) 259) process_yaml_item(i)`
Original 2010-09-03 version David Blume authored 6 years ago	`260) 261) else: 262) if hasattr(feed, 'bozo_exception'): 263) e = feed.bozo_exception`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`264) if isinstance(e, urllib2.URLError):`
Original 2010-09-03 version David Blume authored 6 years ago	`265) print_last_line = True 266) if hasattr(e, 'reason'): 267) if e.reason[0] == 110: 268) print "the feed's connection timed out." 269) print_last_line = False 270) elif e.reason[0] == 111: 271) print "the feed's connection was refused." 272) print_last_line = False 273) elif e.reason[0] == 104: 274) print "the feed reset the connection." 275) print_last_line = False 276) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`277) print "the feed had a URLError with reason %s." % (str(e.reason),)`
Original 2010-09-03 version David Blume authored 6 years ago	`278) print_last_line = False 279) if print_last_line:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`280) print "the feed had a URLError %s" % (str(e),) 281) elif isinstance(e, httplib.BadStatusLine): 282) print "the feed gave a bad status line. (%s)" % (str(e),)`
Original 2010-09-03 version David Blume authored 6 years ago	`283) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`284) if len(str(e)): 285) print "the feed bozo_exception: %s \"%s\"" % (str(e.__class__), str(e))`
Original 2010-09-03 version David Blume authored 6 years ago	`286) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`287) print "the feed bozo_exception: %s %s" % (str(e.__class__), repr(e))`
Original 2010-09-03 version David Blume authored 6 years ago	`288) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`289) print "the feed returned class %s, %s" % (str(feed.__class__), str(feed))`
Original 2010-09-03 version David Blume authored 6 years ago	`290)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`291)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`292) def process_item(feed_item, yaml_items):`
Catch up to production agai... David Blume authored 6 years ago	`293) """Processes an RSS feed item, and converts it to a YAML item"""`
Original 2010-09-03 version David Blume authored 6 years ago	`294) # Get the time 295) global any_entry_added`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`296) timecode_now = int(time.time())`
Original 2010-09-03 version David Blume authored 6 years ago	`297) date_parsed = time.gmtime()`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`298) if hasattr(feed_item, 'issued_parsed'):`
Original 2010-09-03 version David Blume authored 6 years ago	`299) date_parsed = feed_item.issued_parsed`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`300) elif hasattr(feed_item, 'date_parsed'):`
Original 2010-09-03 version David Blume authored 6 years ago	`301) date_parsed = feed_item.date_parsed 302) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`303) print "process_item found no timestamp for", asciiize(feed_item.link) 304) timecode_parsed = calendar.timegm(date_parsed)`
Original 2010-09-03 version David Blume authored 6 years ago	`305)`
Catch up to production agai... David Blume authored 6 years ago	`306) link = feed_item.link 307) if hasattr(feed_item, 'feedburner_origlink'): 308) link = feed_item.feedburner_origlink 309) 310) # TODO 2018-01-18: Leave in the ncid for URL clicks, but remove during processing. 311) # suffix_to_remove = '?ncid=rss' 312) # if link.endswith(suffix_to_remove): 313) # link = link[:-len(suffix_to_remove)] 314)`
Original 2010-09-03 version David Blume authored 6 years ago	`315) # Look for i.feedburner_origlink in yaml_items 316) yaml_item = None 317) for i in yaml_items:`
Catch up to production agai... David Blume authored 6 years ago	`318) if link == i['link']:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`319) yaml_item = i 320) break`
Catch up to production agai... David Blume authored 6 years ago	`321) if yaml_item is None:`
Original 2010-09-03 version David Blume authored 6 years ago	`322) author = ''`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`323) if hasattr(feed_item, 'author'): 324) author = asciiize(feed_item.author)`
Original 2010-09-03 version David Blume authored 6 years ago	`325)`
Catch up to production agai... David Blume authored 6 years ago	`326) # Make a new yaml_item`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`327) yaml_item = {'title' : asciiize(feed_item.title), 328) 'link' : asciiize(link), 329) 'author' : author, 330) 'tags' : [], 331) 'orig_posted' : timecode_parsed, 332) 'qualified' : -1, 333) 'comment_times' : [], 334) 'fb_comments' : [], 335) 'fb_shares' : [], 336) 'fb_likes' : [], 337) 'slash_comment_times' : [], 338) 'slash_comments' : []`
Original 2010-09-03 version David Blume authored 6 years ago	`339) }`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`340) if hasattr(feed_item, 'tags'):`
Original 2010-09-03 version David Blume authored 6 years ago	`341) for i in feed_item.tags:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`342) yaml_item['tags'].append(asciiize(i.term))`
Original 2010-09-03 version David Blume authored 6 years ago	`343)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`344) yaml_items.insert(0, yaml_item)`
Original 2010-09-03 version David Blume authored 6 years ago	`345) any_entry_added = True 346) 347) # Maybe check to ensure that this item isn't too old. 348) if timecode_parsed < timecode_now - 60 * 30 * 9: 349) return 350) 351) # Now, add the new values`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`352) if hasattr(feed_item, 'slash_comments') and len(yaml_item['slash_comments']) < 8:`
Original 2010-09-03 version David Blume authored 6 years ago	`353) any_entry_added = True`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`354) yaml_item['slash_comment_times'].append(timecode_now) 355) yaml_item['slash_comments'].append(int(feed_item.slash_comments))`
Original 2010-09-03 version David Blume authored 6 years ago	`356)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`357)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`358) def process_yaml_item(yaml_item):`
Original 2010-09-03 version David Blume authored 6 years ago	`359) global any_entry_added 360)`
Catch up to production agai... David Blume authored 6 years ago	`361) # Related to TODO 2018-01-18: Remove ncid only during processing. 362) link = yaml_item['link'] 363) suffix_to_remove = '?ncid=rss' 364) # Maybe we should find() it instead, in case feedburner adds other options 365) if link.endswith(suffix_to_remove): 366) link = link[:-len(suffix_to_remove)] 367)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`368) timecode_now = int(time.time()) 369) if len(yaml_item['fb_comments']) < 8:`
Catch up to production agai... David Blume authored 6 years ago	`370) num_shares, num_comments, num_likes = Get_fb_stats(link)`
Original 2010-09-03 version David Blume authored 6 years ago	`371) if num_comments != -1: 372) any_entry_added = True`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`373) yaml_item['comment_times'].append(timecode_now) 374) yaml_item['fb_shares'].append(num_shares) 375) yaml_item['fb_comments'].append(num_comments) 376) yaml_item['fb_likes'].append(num_likes)`
Original 2010-09-03 version David Blume authored 6 years ago	`377)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`378) # if len(yaml_item['reddit_']) < 8:`
Better conformance to PEP-8... David Blume authored 6 years ago	`379) # num_ = get_reddit_stats(link)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`380) # if num_ != -1: 381) # any_entry_added = True 382) # yaml_item['reddit_times'].append(timecode_now) 383) # yaml_item['reddit_'].append(num_)`
Original 2010-09-03 version David Blume authored 6 years ago	`384)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`385)`
Better conformance to PEP-8... David Blume authored 6 years ago	`386) def get_reddit_stats(url_string):`
Catch up to production agai... David Blume authored 6 years ago	`387) """ Consider curl "https://www.reddit.com/api/info.json?url=http://i.imgur.com/HG9dJ.jpg"`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`388) """`
Original 2010-09-03 version David Blume authored 6 years ago	`389) return -1 390)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`391)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`392) def Get_fb_stats(url_string):`
Moved SMTP credentials to s... David Blume authored 6 years ago	`393) """Use graph's "engagement" field to get reactions and shares."""`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`394) shares = -1 395) comments = -1`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`396) likes = -1 397)`
Catch up to production agai... David Blume authored 6 years ago	`398) url_string = url_string.encode('utf-8') 399)`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`400) try:`
Catch up to production agai... David Blume authored 6 years ago	`401) encoded = urllib.urlencode({'access_token': facebook_token}) 402) url = 'https://graph.facebook.com/v2.11/?id=%s&fields=engagement&%s' 403) f = urllib2.urlopen(url % (urllib.quote_plus(url_string), encoded))`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`404) data = f.read() 405) f.close()`
Catch up to production agai... David Blume authored 6 years ago	`406) except (urllib2.URLError, httplib.BadStatusLine) as e:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`407) if hasattr(e, 'reason'): # URLError`
Catch up to production agai... David Blume authored 6 years ago	`408) if hasattr(e, 'code'): 409) print "Get_fb_stats got an error (1):", e.code, e.reason, url_string 410) else: 411) print "Get_fb_stats got an error (2):", e.reason, url_string`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`412) elif hasattr(e, 'code'): #URLError`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`413) print "Get_fb_stats got an error. Code:", e.code, url_string 414) else:`
Catch up to production agai... David Blume authored 6 years ago	`415) print "Get_fb_stats got an error (3):", str(e) 416) return shares, comments, likes`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`417) if len(data) > 20:`
Catch up to production agai... David Blume authored 6 years ago	`418) d = json.loads(data)['engagement'] 419) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`420) shares = d['share_count']`
Catch up to production agai... David Blume authored 6 years ago	`421) except KeyError:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`422) shares = 0`
Catch up to production agai... David Blume authored 6 years ago	`423) 424) try: 425) likes = d['reaction_count'] 426) except KeyError: 427) likes = 0 428) 429) # TODO 2018-01-18: og_object metric was likes + shares + comments 430) # Here we'll combine likes and shares, and comments with plugin_comments 431) likes += shares 432) 433) try: 434) comments = d['comment_plugin_count'] + d['comment_count'] 435) except KeyError: 436) comments = 0`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`437) else: 438) print "Get_fb_stats got too little data for ", url_string`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`439) return shares, comments, likes`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`440) 441)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`442) def make_index_html(yaml_items, weekend_stats, weekday_stats): 443) """Writes a static index.html file from the YAML items."""`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`444) cur_time = int(time.time()) 445) new_index_fullpath = os.path.join(localdir, 'index.html_new') 446) index_fullpath = os.path.join(localdir, 'index.html')`
Original 2010-09-03 version David Blume authored 6 years ago	`447)`
Google terminated image cha... David Blume authored 5 years ago	448) chart_io = cStringIO.StringIO() 449) for image_index, image in enumerate(yaml_items[:40]): 450) tag_hit = False 451) if image['author'].lower() in authors_to_post: 452) tag_hit = True 453) elif len(set([j.lower() for j in image['tags']]) & tags_to_post) > 0: 454) tag_hit = True 455) write_chart_data(image['orig_posted'], 456) image['comment_times'], 457) image['fb_comments'], 458) image[rhs_metric_times], 459) image[rhs_metric], 460) image['qualified'], 461) image_index, 462) tag_hit, 463) chart_io 464) )
Catch up to production agai... David Blume authored 6 years ago	`465)`
Google terminated image cha... David Blume authored 5 years ago	`466) with codecs.open(new_index_fullpath, 'w', 'utf-8') as f:`
Fix comment, and update a URL. David Blume authored 5 years ago	`467) f.write(html_head % (even_background, odd_background, img_width, chart_io.getvalue()))`
Google terminated image cha... David Blume authored 5 years ago	`468) chart_io.close()`
Upgrade yaml module dblume authored 5 months ago	469) f.write('<div align="center">\n<table class="legend">\n<tr><th></th><th>Median</th><th>Mean</th><th>Std. Dev</th><th>Threshold (mean + sigma * %1.1f)</th></tr>\n' % threshold_sigma_factor) 470) f.write('<tr><th>Weekday</th><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td></tr>\n' % (weekday_stats[2][0], weekday_stats[2][1], weekday_stats[2][2], weekday_stats[2][1] + weekday_stats[2][2] * threshold_sigma_factor)) 471) f.write('<tr><th>Weekend</th><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td></tr>\n' % (weekend_stats[2][0], weekend_stats[2][1], weekend_stats[2][2], weekend_stats[2][1] + weekend_stats[2][2] * threshold_sigma_factor))
Catch up to production agai... David Blume authored 6 years ago	`472) f.write('</table></div>\n<br />\n') 473) f.write('<div align="center">\n<table>\n') 474) for image_index, image in enumerate(yaml_items[:40]):`
Fix comment, and update a URL. David Blume authored 5 years ago	`475) f.write('<tr valign="center" class="%s">\n <td><strong><a href="%s">%s</a></strong> <span class="date">at %s</span> <span class="author">by %s</span></td>\n' % \`
Catch up to production agai... David Blume authored 6 years ago	`476) (image_index % 2 and "even" or "odd", 477) image['link'], 478) image['title'].encode('ascii', 'xmlcharrefreplace'),`
Fix comment, and update a URL. David Blume authored 5 years ago	`479) time.strftime("%H:%M", time.localtime(image['orig_posted'])).encode('ascii', 'xmlcharrefreplace'),`
Catch up to production agai... David Blume authored 6 years ago	`480) image['author'].encode('ascii', 'xmlcharrefreplace'), 481) ) 482) ) 483) f.write(' <td>%s<td>\n' % (image['qualified'] != -1 and '<img src="star_30.png" width="30" height="29" />' or ''))`
Google terminated image cha... David Blume authored 5 years ago	`484) f.write(' <td><div id="chart%d" /></td></tr>\n' % (image_index, ))`
Catch up to production agai... David Blume authored 6 years ago	`485) f.write(html_footer) 486)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`487) if os.path.exists(index_fullpath): 488) os.unlink(index_fullpath) 489) shutil.move(new_index_fullpath, index_fullpath)`
Original 2010-09-03 version David Blume authored 6 years ago	`490)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`491)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`492) def make_feed_file(yaml_items): 493) """Writes the RSS feed file with the YAML items."""`
Catch up to production agai... David Blume authored 6 years ago	`494) with codecs.open(os.path.join(localdir, 'rss_feed.xml'), 'wb', 'utf-8') as f:`
Fix comment, and update a URL. David Blume authored 5 years ago	`495) f.write("<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<rss version=\"2.0\" xmlns:atom=\"http://www.w3.org/2005/Atom\">\n<channel>\n<atom:link href=\"http://techcrunch.dlma.com/rss_feed.xml\" rel=\"self\" type=\"application/rss+xml\"/>\n<title>Trending at TechCrunch</title><link>http://techcrunch.dlma.com</link>")`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`496) f.write("<pubDate>%s</pubDate><description>Automatically Generated Feed</description><language>en-us</language>\n" % (time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`497) count = 0 498) for item in yaml_items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`499) now = time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime(item['orig_posted']))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`500) if item['qualified'] != -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`501) escaped_title = cgi.escape(item['title']).encode('ascii', 'xmlcharrefreplace') 502) escaped_author = cgi.escape(item['author']).encode('ascii', 'xmlcharrefreplace') 503) f.write("<item><title>%s</title><pubDate>%s</pubDate><link>%s</link><guid isPermaLink=\"false\">%s</guid><description><![CDATA[By: %s]]></description></item>\n" % \ 504) (escaped_title, now, item['link'], item['link'], escaped_author))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`505) count += 1 506) if count > 14: 507) break`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`508) f.write("</channel></rss>")`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`509)`
Original 2010-09-03 version David Blume authored 6 years ago	`510) 511) if __name__=='__main__': 512) start_time = time.time() 513) progress_text = [] 514) 515) old_stdout = sys.stdout 516) old_stderr = sys.stderr`
Little cleanup, div to rese... David Blume authored 5 years ago	`517) sys.stdout = sys.stderr = cStringIO.StringIO()`
Original 2010-09-03 version David Blume authored 6 years ago	`518) 519) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`520) localdir = os.path.abspath(os.path.dirname(sys.argv[0]))`
Original 2010-09-03 version David Blume authored 6 years ago	`521) # 522) # Read in techcrunch.yaml 523) # 524) # [ { 'title' : 'Title Text', 525) # 'link' : u'http://techcrunch.com/2010/08/17/google-buzz-who-to-follow/', 526) # 'author' : u'MG Siegler', 527) # 'orig_posted' : 1282197199 528) # 'tags' : [ u'Google', u'privacy' ] 529) # 'qualified' : -1 530) # 'comment_times' : [ 1282197199, 1282197407 ]`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`531) # 'fb_comments' : [ 0, 5 ] 532) # 'fb_shares' : [ 0, 300 ] 533) # 'fb_likes' : [ 0, 19 ]`
Original 2010-09-03 version David Blume authored 6 years ago	`534) # 'slash_comment_times' : [ 1282197199, 1282197407 ] 535) # 'slash_comments' : [ 0, 5 ] 536) # }, 537) # { ... } 538) # ] 539) #`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`540) yaml_fullpath = os.path.join(localdir, 'techcrunch.yaml') 541) if os.path.exists(yaml_fullpath): 542) with open(yaml_fullpath, 'rb') as f:`
Upgrade yaml module dblume authored 5 months ago	`543) items = yaml.load(f, Loader=yaml.Loader)`
Catch up to production agai... David Blume authored 6 years ago	`544) if items is None: 545) print yaml_fullpath, "exists, but was empty." 546) items = []`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`547) 548) # Do any dictionary item updating that might be necessary 549) # for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`550) # if not item.has_key('fb_shares'):`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`551) # item['fb_shares'] = []`
Original 2010-09-03 version David Blume authored 6 years ago	`552) else: 553) print "could not open", yaml_fullpath 554) items = [] 555)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`556) with open(os.path.join(localdir, 'facebook-token.txt'), 'r') as f:`
New feed URL, added comment... dblume authored 2 years ago	`557) json_obj = json.load(f) 558) facebook_token = json_obj['access_token']`
Catch up to production agai... David Blume authored 6 years ago	`559)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`560) progress_text = ["read techcrunch.yaml"] 561) process_feed(items)`
Original 2010-09-03 version David Blume authored 6 years ago	`562) 563) # 564) # If any work was done, then write files. 565) #`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`566) if any_entry_added:`
Better conformance to PEP-8... David Blume authored 6 years ago	`567) weekend_stats, weekday_stats = analysis.process_feed(items, rhs_metric, rhs_metric_times)`
Original 2010-09-03 version David Blume authored 6 years ago	`568)`
Fix comment, and update a URL. David Blume authored 5 years ago	`569) # We'll only look at the stats up to 2 hours after posting.`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`570) weekend_median, weekend_mean, weekend_sigma = weekend_stats[2]`
Upgrade yaml module dblume authored 5 months ago	`571) weekend_threshold = weekend_mean + weekend_sigma * threshold_sigma_factor`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`572) weekday_median, weekday_mean, weekday_sigma = weekday_stats[2]`
Upgrade yaml module dblume authored 5 months ago	`573) weekday_threshold = weekday_mean + weekday_sigma * threshold_sigma_factor`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`574) for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`575) wday = time.localtime(item['orig_posted']).tm_wday`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`576) if wday == 5 or wday == 6: 577) threshold = weekend_threshold 578) else: 579) threshold = weekday_threshold 580) if item['qualified'] == -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`581) for i in range(len(item[rhs_metric_times])): 582) r_time = item[rhs_metric_times][i]`
Fix comment, and update a URL. David Blume authored 5 years ago	`583) if r_time - item['orig_posted'] < 7200:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`584) if item[rhs_metric][i] >= threshold:`
New feed URL, added comment... dblume authored 2 years ago	`585) # Comment out when graph.facebook.com engagement returns only 0s.`
2011-02-04: Update to the c... David Blume authored 6 years ago	`586) item['qualified'] = threshold`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`587) break`
Fix comment, and update a URL. David Blume authored 5 years ago	`588) else: 589) break`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`590) 591) # Automatically add those items whose authors and tags I like 592) for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`593) if item['qualified'] == -1 and len(item[rhs_metric_times]) > 0:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`594) if item['author'].lower() in authors_to_post:`
2011-02-04: Update to the c... David Blume authored 6 years ago	`595) item['qualified'] = threshold`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`596) elif len(set([j.lower() for j in item['tags']]) & tags_to_post) > 0:`
2011-02-04: Update to the c... David Blume authored 6 years ago	`597) item['qualified'] = threshold`
Original 2010-09-03 version David Blume authored 6 years ago	`598) 599) # 600) # Write out the updated yaml file. 601) #`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`602) 603) # For the one file we really use, write to a file on the side, then move it.`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`604) yaml_newfile_fullpath = os.path.join(localdir, 'techcrunch_temp_writable.yaml') 605) with open(yaml_newfile_fullpath, 'wb') as f:`
Upgrade yaml module dblume authored 5 months ago	`606) yaml.dump(items, f, default_flow_style=None, width=120)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`607) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`608) os.rename(yaml_newfile_fullpath, yaml_fullpath)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`609) except OSError as e: 610) print "The source file was", yaml_newfile_fullpath, "and exists =", os.path.isfile(yaml_newfile_fullpath)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`611) with open(os.path.join(localdir, 'techcrunch_text.yaml'), 'w') as f:`
Upgrade yaml module dblume authored 5 months ago	`612) yaml.dump(items, f, default_flow_style=None, width=120)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`613) with codecs.open(os.path.join(localdir, 'techcrunch_unicode.yaml'), 'w', 'utf-8') as f:`
Upgrade yaml module dblume authored 5 months ago	`614) yaml.dump(items, f, default_flow_style=None, encoding='utf-8', width=120)`
Original 2010-09-03 version David Blume authored 6 years ago	`615)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`616) make_feed_file(items)`
Original 2010-09-03 version David Blume authored 6 years ago	`617)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`618) make_index_html(items, weekend_stats, weekday_stats)`
Original 2010-09-03 version David Blume authored 6 years ago	`619) else: 620) print "No entries were added this time." 621)`
Catch up to production agai... David Blume authored 6 years ago	`622) except Exception as e:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`623) exceptional_text = "An exception occurred: " + str(e.__class__) + " " + str(e) 624) print exceptional_text, ' '.join(progress_text) 625) traceback.print_exc(file=sys.stdout)`
Original 2010-09-03 version David Blume authored 6 years ago	`626) try:`
Replace hardcoded string wi... David Blume authored 6 years ago	`627) send_email('Exception thrown in ' + os.path.basename(__file__),`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`628) exceptional_text + "\n" + traceback.format_exc(),`
Replace hardcoded string wi... David Blume authored 6 years ago	`629) (smtp_creds.default_recipient,))`
Catch up to production agai... David Blume authored 6 years ago	`630) except Exception as e:`
Original 2010-09-03 version David Blume authored 6 years ago	`631) print "Could not send email to notify you of the exception. :(" 632) 633) message = sys.stdout.getvalue() 634) sys.stdout = old_stdout 635) sys.stderr = old_stderr 636) if not debug: 637) print message 638) 639) # Finally, let's save this to a statistics page`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`640) if os.path.exists(os.path.join(localdir, 'stats.txt')): 641) with open(os.path.join(localdir, 'stats.txt')) as f:`
Original 2010-09-03 version David Blume authored 6 years ago	`642) lines = f.readlines() 643) else: 644) lines = []`
Catch up to production agai... David Blume authored 6 years ago	`645) lines = lines[:672] # Just keep the past week's worth`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`646) # status = len(message.strip()) and message.strip().replace('\n', ' - ') or "OK" 647) status = len(message.strip()) and '\n '.join( message.splitlines()) or "OK" 648) lines.insert(0, "%s %3.0fs %s\n" % (time.strftime('%Y-%m-%d, %H:%M', time.localtime()), time.time() - start_time, status)) 649) with open(os.path.join(localdir,'stats.txt' ), 'w') as f: 650) f.writelines(lines)`