techcrunch.py (dc494fd) - techcrunch.git

dc494fdab54d2cd058f27ed9f6502416dc02a92a

Upgrade to Python3 dblume authored 5 months ago	`1) #!/home/dblume/opt/python-3.9.6/bin/python3`
Original 2010-09-03 version David Blume authored 6 years ago	`2) #`
Moved SMTP credentials to s... David Blume authored 6 years ago	`3) # Testing without affecting the yaml file and saving the updated one aside: 4) # cp techcrunch.yaml techcrunch.yaml_back; ./techcrunch.py; \ 5) # cp techcrunch.yaml techcrunch_tmp.yaml; cp techcrunch.yaml_back techcrunch.yaml`
Original 2010-09-03 version David Blume authored 6 years ago	`6) import feedparser 7) import sys`
Upgrade yaml module dblume authored 5 months ago	`8) if sys.path[0] != '': 9) sys.path.insert(0, '') # DXB Only needed for cronjobs to find yaml 10) import yaml`
Original 2010-09-03 version David Blume authored 6 years ago	`11) import os 12) import time 13) import codecs 14) import traceback 15) import calendar 16) import pickle`
Upgrade to Python3 dblume authored 5 months ago	`17) import urllib.request, urllib.parse, urllib.error 18) import http.client`
Original 2010-09-03 version David Blume authored 6 years ago	`19) import shutil 20) import smtplib 21) import analysis`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`22) import json`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`23) import xml`
2011-02-04: Update to the c... David Blume authored 6 years ago	`24) import operator`
Upgrade to Python3 dblume authored 5 months ago	`25) import html 26) import io`
Moved SMTP credentials to s... David Blume authored 6 years ago	`27) import smtp_creds # Your own credentials, used in send_email()`
Original 2010-09-03 version David Blume authored 6 years ago	`28) 29) debug = True 30) any_entry_added = False`
Google terminated image cha... David Blume authored 5 years ago	`31) tags_to_post = {'apple', 'google', 'roku'}`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`32) authors_to_post = ['michael arrington',] 33)`
Upgrade yaml module dblume authored 5 months ago	`34) # 2022-02-24: Seeing too many posts with just mean + sigma, adjust with factor 35) threshold_sigma_factor = 1.5 36)`
Catch up to production agai... David Blume authored 6 years ago	`37) # TODO 2018-01-18: Maybe combine fb_likes with bf_shares or something... 38) rhs_metric = 'fb_likes'`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`39) rhs_metric_times = 'comment_times'`
Original 2010-09-03 version David Blume authored 6 years ago	`40) 41) localdir = '' 42)`
Google terminated image cha... David Blume authored 5 years ago	`43) html_head = """<!DOCTYPE html> 44) <html><head>`
HTML page needs to be UTF-8 dblume authored 5 months ago	`45) <meta charset="UTF-8">`
Original 2010-09-03 version David Blume authored 6 years ago	`46) <title>TechCrunch Feed Filter</title> 47) <!-- <link rel="alternate" type="application/rss+xml" title="RSS feed" href="http://techcrunch.dlma.com/rss_feed.xml" /> --> 48) <link rel="alternate" type="application/rss+xml" title="RSS feed" href="http://feeds.feedburner.com/TrendingAtTechcrunch" /> 49) <style type="text/css"> 50) body { font-family: "Arial", san-serif; }`
Fix comment, and update a URL. David Blume authored 5 years ago	`51) .author { font-size: smaller; color:gray; } 52) .date { font-size: smaller; color:gray; }`
Original 2010-09-03 version David Blume authored 6 years ago	53) .h3 { font-size: larger; } 54) a { text-decoration: none; } 55) /* table { border: none; border-collapse:collapse; font-size: large } */ 56) table { border-collapse: collapse; } 57) table.legend { border:1px solid LightSlateGray; font-size: medium; border-collapse:separated; } 58) table.legend th { border: 1px solid LightSlateGray; background-color: #E0E0E0; } 59) table.legend td { border: 1px solid LightSlateGray; } 60) tr.even { background:#%s; padding: 2em; } 61) tr.odd { background:#%s; padding-bottom: 2em; }
Little cleanup, div to rese... David Blume authored 5 years ago	`62) td div { height: 68px; }`
Original 2010-09-03 version David Blume authored 6 years ago	`63) </style>`
Google terminated image cha... David Blume authored 5 years ago	`64) <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script> 65) <script type="text/javascript"> 66) google.charts.load('current', {'packages':['corechart']}); 67) google.charts.setOnLoadCallback(drawChart); 68) function drawChart() { 69) var options = {`
Fix comment, and update a URL. David Blume authored 5 years ago	`70) width:%d,`
Google terminated image cha... David Blume authored 5 years ago	`71) height:68, 72) pointSize:0.1, 73) dataOpacity:1.0, 74) series: { 0: {targetAxisIndex:0}, 1: {targetAxisIndex:1, color:'limegreen'} }, 75) vAxis: { gridlines: {count: 0}, maxValue: 1 }, 76) hAxis: { gridlines: {count: 0}, ticks: [] }, 77) vAxes: { 0: {textStyle: {fontSize: 11, color: 'blue'} }, 1: {viewWindowMode: 'maximized', baselineColor: '#A0D0A0', textStyle: {fontSize: 11, color: 'limegreen'} } }, 78) }; 79) %s 80) } 81) </script> 82) </head> 83) <body>`
Original 2010-09-03 version David Blume authored 6 years ago	`84) <div align='center'><h3>TechCrunch Feed Filter</h3></div>`
Add feed icon and link to t... David Blume authored 6 years ago	`85) This page shows what analysis is done to filter the noise away from the Techcrunch feed into 86) <a href="http://feeds.feedburner.com/TrendingAtTechcrunch"> a more concise feed <img src="feed.png" alt="feed" height="14" width="14"></a>.`
Google terminated image cha... David Blume authored 5 years ago	`87) <a href="https://david.dlma.com/blog/my-techcrunch-feed-filter">Learn more about the Feed Filter</a>.<br /><br />`
Original 2010-09-03 version David Blume authored 6 years ago	`88) """ 89) 90) html_footer = """ 91) </table> 92) </div><br /> 93) <div align='center'>Thanks to <a href="http://www.feedparser.org/">The Universal Feed Parser module</a>,`
Add feed icon and link to t... David Blume authored 6 years ago	`94) <a href="http://pyyaml.org/">PyYAML</a> and <a href="http://code.google.com/apis/chart/">Google Charts</a>.<br />`
Google terminated image cha... David Blume authored 5 years ago	95) <a href="http://git.dlma.com/techcrunch.git/">source</a> • <a href="techcrunch.yaml">raw data</a> • <a href="stats.txt">status</a><br />© 2011 <a href="https://david.dlma.com">David Blume</a></div><br /> 96) </body> 97) </html> 98) """ 99) 100) chart_data_header = """ var data = google.visualization.arrayToDataTable([ 101) ['', 'Comments', 'Shares', {'type': 'string', 'role': 'style'}], 102) """ 103) chart_data_middle = """ ]); 104) var chart = new google.visualization.LineChart(document.getElementById('chart%d')); 105) options.backgroundColor = '#%s';
Original 2010-09-03 version David Blume authored 6 years ago	`106) """ 107)`
Fix comment, and update a URL. David Blume authored 5 years ago	`108) img_width = 240`
Google terminated image cha... David Blume authored 5 years ago	`109) img_height = 68`
Original 2010-09-03 version David Blume authored 6 years ago	`110) 111) series_1_color = "0000FF" 112) series_2_color = "00AA00" 113) threshold_color = "FF8C00"`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`114) tag_color = "F01000"`
Original 2010-09-03 version David Blume authored 6 years ago	`115) 116) even_background = "F8F8F8" 117) odd_background = "E8E8E8" 118)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`119) even_watermark = "E0E0FF" 120) odd_watermark = "D0D0F0" 121) 122)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`123) def send_email(subject, message, toaddrs, 124) fromaddr='"%s" <%s>' % (os.path.basename(__file__), smtp_creds.user)): 125) """ Sends Email """ 126) smtp = smtplib.SMTP(smtp_creds.server, port=smtp_creds.port) 127) smtp.login(smtp_creds.user, smtp_creds.passw)`
Better conformance to PEP-8... David Blume authored 6 years ago	`128) smtp.sendmail(fromaddr, 129) toaddrs,`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`130) "Content-Type: text/plain; charset=\"us-ascii\"\r\nFrom: %s\r\nTo: %s\r\nSubject: %s\r\n%s" % \ 131) (fromaddr, ", ".join(toaddrs), subject, message))`
Original 2010-09-03 version David Blume authored 6 years ago	`132) smtp.quit() 133)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`134)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`135) def index_id(a_list, op, elem):`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`136) try:`
Upgrade to Python3 dblume authored 5 months ago	`137) return next((index for index, item in enumerate(a_list) if op(item, elem)))`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`138) except: 139) return -1 140)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`141)`
Google terminated image cha... David Blume authored 5 years ago	`142) def write_chart_data(time_posted, lhs_times, lhs_values, rhs_times, 143) rhs_values, threshold_value, image_index, tag_hit, chart_io):`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`144) # lhs_times, lhs_values = zip(comments) 145) # rhs_times, rhs_values = zip(rhs)`
Google terminated image cha... David Blume authored 5 years ago	`146) is_odd_row = image_index % 2`
Original 2010-09-03 version David Blume authored 6 years ago	`147)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`148) if not len(lhs_times): 149) lhs_times = [time_posted,] 150) if not len(lhs_values): 151) lhs_values = [0,] 152) if not len(rhs_times): 153) rhs_times = [time_posted,] 154) if not len(rhs_values): 155) rhs_values = [0,] 156) 157) lhs_times = [(i - time_posted) / 1800 for i in lhs_times] 158) rhs_times = [(i - time_posted) / 1800 for i in rhs_times] 159)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`160) met_threshold_pt = -1 161) if threshold_value != -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`162) met_threshold_pt = index_id(rhs_values, operator.ge, threshold_value)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`163) if met_threshold_pt == -1 or tag_hit: 164) # This can happen if threshold_value was set to a number 165) # because the author or a tag was matched, but the article 166) # was unpopular. We choose to put a marker at point index 0. 167) met_threshold_pt = 0 168) 169) if is_odd_row != 0: 170) bg_color = even_background 171) else: 172) bg_color = odd_background`
Google terminated image cha... David Blume authored 5 years ago	`173) 174) chart_io.write(chart_data_header) 175) for i in range(8): 176) if i == met_threshold_pt: 177) if tag_hit: 178) style = "'point { size: 5; fill-color: #FF0000; shape-type: diamond}'" 179) else: 180) style = "'point { size: 5; fill-color: #FF8C00; }'" 181) else: 182) style = "null" 183) if i < len(lhs_values): 184) lhs_value = str(lhs_values[i]) 185) else: 186) lhs_value = "null" 187) if i < len(rhs_values): 188) rhs_value = str(rhs_values[i])`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`189) else:`
Google terminated image cha... David Blume authored 5 years ago	`190) rhs_value = "null" 191) chart_io.write(" [%d, %s, %s, %s],\n" % (i, lhs_value, rhs_value, style)) 192) chart_io.write(chart_data_middle % (image_index, bg_color)) 193) if met_threshold_pt == -1 and not tag_hit: 194) chart_io.write(" delete options.vAxes[1].baseline;\n") 195) else: 196) chart_io.write(" options.vAxes[1].baseline = %d;\n" % (threshold_value,)) 197) chart_io.write(" chart.draw(data, options);\n\n")`
Original 2010-09-03 version David Blume authored 6 years ago	`198)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`199)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`200) def process_feed(yaml_items):`
Moved SMTP credentials to s... David Blume authored 6 years ago	`201) """Retrieve the url and process it.`
Original 2010-09-03 version David Blume authored 6 years ago	`202) feed_info (in, out) A tuple that describes an individual feed, like its name and etag. 203) """`
Upgrade yaml module dblume authored 5 months ago	`204) #feed = feedparser.parse('https://techcrunch.com/feed/') 205) feed = feedparser.parse('https://pi.dlma.com/techcrunch_feed.php') # DXB temporary until removed from denylist`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`206) if hasattr(feed, 'status'):`
Original 2010-09-03 version David Blume authored 6 years ago	`207) if feed.status == 304: 208) pass 209) else: 210) if feed.status != 200 and feed.status != 307 and feed.status != 301 and feed.status != 302: 211) if feed.status == 503:`
Upgrade to Python3 dblume authored 5 months ago	`212) print("the feed is temporarily unavailable.")`
Original 2010-09-03 version David Blume authored 6 years ago	`213) elif feed.status == 400:`
Upgrade to Python3 dblume authored 5 months ago	`214) print("the feed says we made a bad request.")`
Original 2010-09-03 version David Blume authored 6 years ago	`215) elif feed.status == 502:`
Upgrade to Python3 dblume authored 5 months ago	`216) print("the feed reported a bad gateway error.")`
Original 2010-09-03 version David Blume authored 6 years ago	`217) elif feed.status == 404:`
Upgrade to Python3 dblume authored 5 months ago	`218) print("the feed says the page was not found.")`
Original 2010-09-03 version David Blume authored 6 years ago	`219) elif feed.status == 500:`
Upgrade to Python3 dblume authored 5 months ago	`220) print("the feed had an internal server error.")`
Original 2010-09-03 version David Blume authored 6 years ago	`221) elif feed.status == 403:`
Upgrade to Python3 dblume authored 5 months ago	`222) print("Access to the feed was forbidden.")`
Original 2010-09-03 version David Blume authored 6 years ago	`223) else:`
Upgrade to Python3 dblume authored 5 months ago	`224) print("the feed returned feed.status %d." % ( feed.status, ))`
Original 2010-09-03 version David Blume authored 6 years ago	`225) else: 226) # Save off this`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`227) if hasattr(feed, 'bozo_exception') and isinstance(feed.bozo_exception, xml.sax._exceptions.SAXParseException):`
Upgrade to Python3 dblume authored 5 months ago	`228) print("Didn't pickle TechCrunch feed because it had a bozo_exception: %s" % (str(feed.bozo_exception)))`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`229) else: 230) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`231) with open(os.path.join(localdir, 'techcrunch_feed.pickle'), 'wb') as f: 232) pickle.dump(feed, f)`
Catch up to production agai... David Blume authored 6 years ago	`233) except(pickle.PicklingError, exceptions.TypeError) as e:`
Upgrade to Python3 dblume authored 5 months ago	`234) print("An error occurred while pickling the feed: %s." % \`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`235) (# str(e.__class__),`
Upgrade to Python3 dblume authored 5 months ago	`236) str(e)))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`237) traceback.print_exc(3, file=sys.stdout)`
Original 2010-09-03 version David Blume authored 6 years ago	`238)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`239) for i in reversed(feed.entries): 240) process_item(i, yaml_items)`
Original 2010-09-03 version David Blume authored 6 years ago	`241) 242) # If we have more than 200 items, remove the old ones.`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`243) while len(yaml_items) > 200:`
Original 2010-09-03 version David Blume authored 6 years ago	`244) yaml_items.pop() 245) 246) for i in yaml_items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`247) process_yaml_item(i)`
Original 2010-09-03 version David Blume authored 6 years ago	`248) 249) else: 250) if hasattr(feed, 'bozo_exception'): 251) e = feed.bozo_exception`
Upgrade to Python3 dblume authored 5 months ago	`252) if isinstance(e, urllib.error.URLError):`
Original 2010-09-03 version David Blume authored 6 years ago	`253) print_last_line = True 254) if hasattr(e, 'reason'):`
HTML page needs to be UTF-8 dblume authored 5 months ago	255) print(f'The feed had a bozo_exception with {e.reason=}') 256) # if hasattr(e, 'reason'): 257) # if e.reason[0] == 110: 258) # print("the feed's connection timed out.") 259) # print_last_line = False 260) # elif e.reason[0] == 111: 261) # print("the feed's connection was refused.") 262) # print_last_line = False 263) # elif e.reason[0] == 104: 264) # print("the feed reset the connection.") 265) # print_last_line = False 266) # else: 267) # print("the feed had a URLError with reason %s." % (str(e.reason),)) 268) # print_last_line = False
Original 2010-09-03 version David Blume authored 6 years ago	`269) if print_last_line:`
HTML page needs to be UTF-8 dblume authored 5 months ago	`270) print(f"the feed had a URLError {str(e)}")`
Upgrade to Python3 dblume authored 5 months ago	`271) elif isinstance(e, http.client.BadStatusLine): 272) print("the feed gave a bad status line. (%s)" % (str(e),))`
Original 2010-09-03 version David Blume authored 6 years ago	`273) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`274) if len(str(e)):`
Upgrade to Python3 dblume authored 5 months ago	`275) print("the feed bozo_exception: %s \"%s\"" % (str(e.__class__), str(e)))`
Original 2010-09-03 version David Blume authored 6 years ago	`276) else:`
Upgrade to Python3 dblume authored 5 months ago	`277) print("the feed bozo_exception: %s %s" % (str(e.__class__), repr(e)))`
Original 2010-09-03 version David Blume authored 6 years ago	`278) else:`
Upgrade to Python3 dblume authored 5 months ago	`279) print("the feed returned class %s, %s" % (str(feed.__class__), str(feed)))`
Original 2010-09-03 version David Blume authored 6 years ago	`280)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`281)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`282) def process_item(feed_item, yaml_items):`
Catch up to production agai... David Blume authored 6 years ago	`283) """Processes an RSS feed item, and converts it to a YAML item"""`
Original 2010-09-03 version David Blume authored 6 years ago	`284) # Get the time 285) global any_entry_added`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`286) timecode_now = int(time.time())`
Original 2010-09-03 version David Blume authored 6 years ago	`287) date_parsed = time.gmtime()`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`288) if hasattr(feed_item, 'issued_parsed'):`
Original 2010-09-03 version David Blume authored 6 years ago	`289) date_parsed = feed_item.issued_parsed`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`290) elif hasattr(feed_item, 'date_parsed'):`
Original 2010-09-03 version David Blume authored 6 years ago	`291) date_parsed = feed_item.date_parsed 292) else:`
Make Python 3 actually work dblume authored 5 months ago	`293) print("process_item found no timestamp for", feed_item.link)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`294) timecode_parsed = calendar.timegm(date_parsed)`
Original 2010-09-03 version David Blume authored 6 years ago	`295)`
Catch up to production agai... David Blume authored 6 years ago	`296) link = feed_item.link 297) if hasattr(feed_item, 'feedburner_origlink'): 298) link = feed_item.feedburner_origlink 299) 300) # TODO 2018-01-18: Leave in the ncid for URL clicks, but remove during processing. 301) # suffix_to_remove = '?ncid=rss' 302) # if link.endswith(suffix_to_remove): 303) # link = link[:-len(suffix_to_remove)] 304)`
Original 2010-09-03 version David Blume authored 6 years ago	`305) # Look for i.feedburner_origlink in yaml_items 306) yaml_item = None 307) for i in yaml_items:`
Catch up to production agai... David Blume authored 6 years ago	`308) if link == i['link']:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`309) yaml_item = i 310) break`
Catch up to production agai... David Blume authored 6 years ago	`311) if yaml_item is None:`
Original 2010-09-03 version David Blume authored 6 years ago	`312) author = ''`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`313) if hasattr(feed_item, 'author'):`
Make Python 3 actually work dblume authored 5 months ago	`314) author = feed_item.author`
Original 2010-09-03 version David Blume authored 6 years ago	`315)`
Catch up to production agai... David Blume authored 6 years ago	`316) # Make a new yaml_item`
Make Python 3 actually work dblume authored 5 months ago	`317) yaml_item = {'title' : feed_item.title, 318) 'link' : link,`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`319) 'author' : author, 320) 'tags' : [], 321) 'orig_posted' : timecode_parsed, 322) 'qualified' : -1, 323) 'comment_times' : [], 324) 'fb_comments' : [], 325) 'fb_shares' : [], 326) 'fb_likes' : [], 327) 'slash_comment_times' : [], 328) 'slash_comments' : []`
Original 2010-09-03 version David Blume authored 6 years ago	`329) }`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`330) if hasattr(feed_item, 'tags'):`
Original 2010-09-03 version David Blume authored 6 years ago	`331) for i in feed_item.tags:`
Make Python 3 actually work dblume authored 5 months ago	`332) yaml_item['tags'].append(i.term)`
Original 2010-09-03 version David Blume authored 6 years ago	`333)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`334) yaml_items.insert(0, yaml_item)`
Original 2010-09-03 version David Blume authored 6 years ago	`335) any_entry_added = True 336) 337) # Maybe check to ensure that this item isn't too old. 338) if timecode_parsed < timecode_now - 60 * 30 * 9: 339) return 340) 341) # Now, add the new values`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`342) if hasattr(feed_item, 'slash_comments') and len(yaml_item['slash_comments']) < 8:`
Original 2010-09-03 version David Blume authored 6 years ago	`343) any_entry_added = True`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`344) yaml_item['slash_comment_times'].append(timecode_now) 345) yaml_item['slash_comments'].append(int(feed_item.slash_comments))`
Original 2010-09-03 version David Blume authored 6 years ago	`346)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`347)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`348) def process_yaml_item(yaml_item):`
Original 2010-09-03 version David Blume authored 6 years ago	`349) global any_entry_added 350)`
Catch up to production agai... David Blume authored 6 years ago	`351) # Related to TODO 2018-01-18: Remove ncid only during processing. 352) link = yaml_item['link'] 353) suffix_to_remove = '?ncid=rss' 354) # Maybe we should find() it instead, in case feedburner adds other options 355) if link.endswith(suffix_to_remove): 356) link = link[:-len(suffix_to_remove)] 357)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`358) timecode_now = int(time.time()) 359) if len(yaml_item['fb_comments']) < 8:`
Catch up to production agai... David Blume authored 6 years ago	`360) num_shares, num_comments, num_likes = Get_fb_stats(link)`
Original 2010-09-03 version David Blume authored 6 years ago	`361) if num_comments != -1: 362) any_entry_added = True`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`363) yaml_item['comment_times'].append(timecode_now) 364) yaml_item['fb_shares'].append(num_shares) 365) yaml_item['fb_comments'].append(num_comments) 366) yaml_item['fb_likes'].append(num_likes)`
Original 2010-09-03 version David Blume authored 6 years ago	`367)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`368) # if len(yaml_item['reddit_']) < 8:`
Better conformance to PEP-8... David Blume authored 6 years ago	`369) # num_ = get_reddit_stats(link)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`370) # if num_ != -1: 371) # any_entry_added = True 372) # yaml_item['reddit_times'].append(timecode_now) 373) # yaml_item['reddit_'].append(num_)`
Original 2010-09-03 version David Blume authored 6 years ago	`374)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`375)`
Better conformance to PEP-8... David Blume authored 6 years ago	`376) def get_reddit_stats(url_string):`
Catch up to production agai... David Blume authored 6 years ago	`377) """ Consider curl "https://www.reddit.com/api/info.json?url=http://i.imgur.com/HG9dJ.jpg"`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`378) """`
Original 2010-09-03 version David Blume authored 6 years ago	`379) return -1 380)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`381)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`382) def Get_fb_stats(url_string):`
Moved SMTP credentials to s... David Blume authored 6 years ago	`383) """Use graph's "engagement" field to get reactions and shares."""`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`384) shares = -1 385) comments = -1`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`386) likes = -1 387)`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`388) try:`
Upgrade to Python3 dblume authored 5 months ago	`389) encoded = urllib.parse.urlencode({'access_token': facebook_token})`
Catch up to production agai... David Blume authored 6 years ago	`390) url = 'https://graph.facebook.com/v2.11/?id=%s&fields=engagement&%s'`
Upgrade to Python3 dblume authored 5 months ago	`391) f = urllib.request.urlopen(url % (urllib.parse.quote_plus(url_string), encoded))`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`392) data = f.read() 393) f.close()`
Upgrade to Python3 dblume authored 5 months ago	`394) except (urllib.error.URLError, http.client.BadStatusLine) as e:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`395) if hasattr(e, 'reason'): # URLError`
Catch up to production agai... David Blume authored 6 years ago	`396) if hasattr(e, 'code'):`
Upgrade to Python3 dblume authored 5 months ago	`397) print("Get_fb_stats got an error (1):", e.code, e.reason, url_string)`
Catch up to production agai... David Blume authored 6 years ago	`398) else:`
Upgrade to Python3 dblume authored 5 months ago	`399) print("Get_fb_stats got an error (2):", e.reason, url_string)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`400) elif hasattr(e, 'code'): #URLError`
Upgrade to Python3 dblume authored 5 months ago	`401) print("Get_fb_stats got an error. Code:", e.code, url_string)`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`402) else:`
Upgrade to Python3 dblume authored 5 months ago	`403) print("Get_fb_stats got an error (3):", str(e))`
Catch up to production agai... David Blume authored 6 years ago	`404) return shares, comments, likes`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`405) if len(data) > 20:`
Catch up to production agai... David Blume authored 6 years ago	`406) d = json.loads(data)['engagement'] 407) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`408) shares = d['share_count']`
Catch up to production agai... David Blume authored 6 years ago	`409) except KeyError:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`410) shares = 0`
Catch up to production agai... David Blume authored 6 years ago	`411) 412) try: 413) likes = d['reaction_count'] 414) except KeyError: 415) likes = 0 416) 417) # TODO 2018-01-18: og_object metric was likes + shares + comments 418) # Here we'll combine likes and shares, and comments with plugin_comments 419) likes += shares 420) 421) try: 422) comments = d['comment_plugin_count'] + d['comment_count'] 423) except KeyError: 424) comments = 0`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`425) else:`
Upgrade to Python3 dblume authored 5 months ago	`426) print("Get_fb_stats got too little data for ", url_string)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`427) return shares, comments, likes`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`428) 429)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`430) def make_index_html(yaml_items, weekend_stats, weekday_stats): 431) """Writes a static index.html file from the YAML items."""`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`432) cur_time = int(time.time()) 433) new_index_fullpath = os.path.join(localdir, 'index.html_new') 434) index_fullpath = os.path.join(localdir, 'index.html')`
Original 2010-09-03 version David Blume authored 6 years ago	`435)`
Upgrade to Python3 dblume authored 5 months ago	`436) chart_io = io.StringIO()`
Google terminated image cha... David Blume authored 5 years ago	`437) for image_index, image in enumerate(yaml_items[:40]): 438) tag_hit = False 439) if image['author'].lower() in authors_to_post: 440) tag_hit = True 441) elif len(set([j.lower() for j in image['tags']]) & tags_to_post) > 0: 442) tag_hit = True 443) write_chart_data(image['orig_posted'], 444) image['comment_times'], 445) image['fb_comments'], 446) image[rhs_metric_times], 447) image[rhs_metric], 448) image['qualified'], 449) image_index, 450) tag_hit, 451) chart_io 452) )`
Catch up to production agai... David Blume authored 6 years ago	`453)`
Google terminated image cha... David Blume authored 5 years ago	`454) with codecs.open(new_index_fullpath, 'w', 'utf-8') as f:`
Fix comment, and update a URL. David Blume authored 5 years ago	`455) f.write(html_head % (even_background, odd_background, img_width, chart_io.getvalue()))`
Google terminated image cha... David Blume authored 5 years ago	`456) chart_io.close()`
Upgrade yaml module dblume authored 5 months ago	457) f.write('<div align="center">\n<table class="legend">\n<tr><th></th><th>Median</th><th>Mean</th><th>Std. Dev</th><th>Threshold (mean + sigma * %1.1f)</th></tr>\n' % threshold_sigma_factor) 458) f.write('<tr><th>Weekday</th><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td></tr>\n' % (weekday_stats[2][0], weekday_stats[2][1], weekday_stats[2][2], weekday_stats[2][1] + weekday_stats[2][2] * threshold_sigma_factor)) 459) f.write('<tr><th>Weekend</th><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td></tr>\n' % (weekend_stats[2][0], weekend_stats[2][1], weekend_stats[2][2], weekend_stats[2][1] + weekend_stats[2][2] * threshold_sigma_factor))
Catch up to production agai... David Blume authored 6 years ago	`460) f.write('</table></div>\n<br />\n') 461) f.write('<div align="center">\n<table>\n') 462) for image_index, image in enumerate(yaml_items[:40]):`
Fix comment, and update a URL. David Blume authored 5 years ago	`463) f.write('<tr valign="center" class="%s">\n <td><strong><a href="%s">%s</a></strong> <span class="date">at %s</span> <span class="author">by %s</span></td>\n' % \`
Catch up to production agai... David Blume authored 6 years ago	`464) (image_index % 2 and "even" or "odd", 465) image['link'],`
Make Python 3 actually work dblume authored 5 months ago	`466) image['title'], 467) time.strftime("%H:%M", time.localtime(image['orig_posted'])), 468) image['author'],`
Catch up to production agai... David Blume authored 6 years ago	`469) ) 470) ) 471) f.write(' <td>%s<td>\n' % (image['qualified'] != -1 and '<img src="star_30.png" width="30" height="29" />' or ''))`
Google terminated image cha... David Blume authored 5 years ago	`472) f.write(' <td><div id="chart%d" /></td></tr>\n' % (image_index, ))`
Catch up to production agai... David Blume authored 6 years ago	`473) f.write(html_footer) 474)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`475) if os.path.exists(index_fullpath): 476) os.unlink(index_fullpath) 477) shutil.move(new_index_fullpath, index_fullpath)`
Original 2010-09-03 version David Blume authored 6 years ago	`478)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`479)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`480) def make_feed_file(yaml_items): 481) """Writes the RSS feed file with the YAML items."""`
Make Python 3 actually work dblume authored 5 months ago	`482) with codecs.open(os.path.join(localdir, 'rss_feed.xml'), 'w', 'utf-8') as f:`
Fix comment, and update a URL. David Blume authored 5 years ago	`483) f.write("<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<rss version=\"2.0\" xmlns:atom=\"http://www.w3.org/2005/Atom\">\n<channel>\n<atom:link href=\"http://techcrunch.dlma.com/rss_feed.xml\" rel=\"self\" type=\"application/rss+xml\"/>\n<title>Trending at TechCrunch</title><link>http://techcrunch.dlma.com</link>")`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`484) f.write("<pubDate>%s</pubDate><description>Automatically Generated Feed</description><language>en-us</language>\n" % (time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`485) count = 0 486) for item in yaml_items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`487) now = time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime(item['orig_posted']))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`488) if item['qualified'] != -1:`
Make Python 3 actually work dblume authored 5 months ago	`489) escaped_title = html.escape(item['title']) 490) escaped_author = html.escape(item['author'])`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`491) f.write("<item><title>%s</title><pubDate>%s</pubDate><link>%s</link><guid isPermaLink=\"false\">%s</guid><description><![CDATA[By: %s]]></description></item>\n" % \ 492) (escaped_title, now, item['link'], item['link'], escaped_author))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`493) count += 1 494) if count > 14: 495) break`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`496) f.write("</channel></rss>")`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`497)`
Original 2010-09-03 version David Blume authored 6 years ago	`498) 499) if __name__=='__main__': 500) start_time = time.time() 501) progress_text = [] 502) 503) old_stdout = sys.stdout 504) old_stderr = sys.stderr`
Upgrade to Python3 dblume authored 5 months ago	`505) sys.stdout = sys.stderr = io.StringIO()`
Original 2010-09-03 version David Blume authored 6 years ago	`506) 507) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`508) localdir = os.path.abspath(os.path.dirname(sys.argv[0]))`
Original 2010-09-03 version David Blume authored 6 years ago	`509) # 510) # Read in techcrunch.yaml 511) # 512) # [ { 'title' : 'Title Text', 513) # 'link' : u'http://techcrunch.com/2010/08/17/google-buzz-who-to-follow/', 514) # 'author' : u'MG Siegler', 515) # 'orig_posted' : 1282197199 516) # 'tags' : [ u'Google', u'privacy' ] 517) # 'qualified' : -1 518) # 'comment_times' : [ 1282197199, 1282197407 ]`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`519) # 'fb_comments' : [ 0, 5 ] 520) # 'fb_shares' : [ 0, 300 ] 521) # 'fb_likes' : [ 0, 19 ]`
Original 2010-09-03 version David Blume authored 6 years ago	`522) # 'slash_comment_times' : [ 1282197199, 1282197407 ] 523) # 'slash_comments' : [ 0, 5 ] 524) # }, 525) # { ... } 526) # ] 527) #`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`528) yaml_fullpath = os.path.join(localdir, 'techcrunch.yaml') 529) if os.path.exists(yaml_fullpath):`
Make Python 3 actually work dblume authored 5 months ago	`530) with open(yaml_fullpath, 'r') as f:`
Upgrade yaml module dblume authored 5 months ago	`531) items = yaml.load(f, Loader=yaml.Loader)`
Catch up to production agai... David Blume authored 6 years ago	`532) if items is None:`
Upgrade to Python3 dblume authored 5 months ago	`533) print(yaml_fullpath, "exists, but was empty.")`
Catch up to production agai... David Blume authored 6 years ago	`534) items = []`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`535) 536) # Do any dictionary item updating that might be necessary 537) # for item in items:`
Make Python 3 actually work dblume authored 5 months ago	`538) # item['link'] = str(item['link'])`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`539) # if not item.has_key('fb_shares'):`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`540) # item['fb_shares'] = []`
Original 2010-09-03 version David Blume authored 6 years ago	`541) else:`
Upgrade to Python3 dblume authored 5 months ago	`542) print("could not open", yaml_fullpath)`
Original 2010-09-03 version David Blume authored 6 years ago	`543) items = [] 544)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`545) with open(os.path.join(localdir, 'facebook-token.txt'), 'r') as f:`
Upgrade to Python3 dblume authored 5 months ago	`546) json_obj = json.load(f) 547) facebook_token = json_obj['access_token']`
Catch up to production agai... David Blume authored 6 years ago	`548)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`549) progress_text = ["read techcrunch.yaml"] 550) process_feed(items)`
Original 2010-09-03 version David Blume authored 6 years ago	`551) 552) # 553) # If any work was done, then write files. 554) #`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`555) if any_entry_added:`
Better conformance to PEP-8... David Blume authored 6 years ago	`556) weekend_stats, weekday_stats = analysis.process_feed(items, rhs_metric, rhs_metric_times)`
Original 2010-09-03 version David Blume authored 6 years ago	`557)`
Fix comment, and update a URL. David Blume authored 5 years ago	`558) # We'll only look at the stats up to 2 hours after posting.`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`559) weekend_median, weekend_mean, weekend_sigma = weekend_stats[2]`
Upgrade yaml module dblume authored 5 months ago	`560) weekend_threshold = weekend_mean + weekend_sigma * threshold_sigma_factor`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`561) weekday_median, weekday_mean, weekday_sigma = weekday_stats[2]`
Upgrade yaml module dblume authored 5 months ago	`562) weekday_threshold = weekday_mean + weekday_sigma * threshold_sigma_factor`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`563) for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`564) wday = time.localtime(item['orig_posted']).tm_wday`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`565) if wday == 5 or wday == 6: 566) threshold = weekend_threshold 567) else: 568) threshold = weekday_threshold 569) if item['qualified'] == -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`570) for i in range(len(item[rhs_metric_times])): 571) r_time = item[rhs_metric_times][i]`
Fix comment, and update a URL. David Blume authored 5 years ago	`572) if r_time - item['orig_posted'] < 7200:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`573) if item[rhs_metric][i] >= threshold:`
New feed URL, added comment... dblume authored 2 years ago	`574) # Comment out when graph.facebook.com engagement returns only 0s.`
2011-02-04: Update to the c... David Blume authored 6 years ago	`575) item['qualified'] = threshold`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`576) break`
Fix comment, and update a URL. David Blume authored 5 years ago	`577) else: 578) break`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`579) 580) # Automatically add those items whose authors and tags I like 581) for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`582) if item['qualified'] == -1 and len(item[rhs_metric_times]) > 0:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`583) if item['author'].lower() in authors_to_post:`
2011-02-04: Update to the c... David Blume authored 6 years ago	`584) item['qualified'] = threshold`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`585) elif len(set([j.lower() for j in item['tags']]) & tags_to_post) > 0:`
2011-02-04: Update to the c... David Blume authored 6 years ago	`586) item['qualified'] = threshold`
Original 2010-09-03 version David Blume authored 6 years ago	`587) 588) # 589) # Write out the updated yaml file. 590) #`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`591) 592) # For the one file we really use, write to a file on the side, then move it.`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`593) yaml_newfile_fullpath = os.path.join(localdir, 'techcrunch_temp_writable.yaml')`
Make Python 3 actually work dblume authored 5 months ago	`594) with open(yaml_newfile_fullpath, 'w') as f:`
Upgrade yaml module dblume authored 5 months ago	`595) yaml.dump(items, f, default_flow_style=None, width=120)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`596) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`597) os.rename(yaml_newfile_fullpath, yaml_fullpath)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`598) except OSError as e:`
Upgrade to Python3 dblume authored 5 months ago	`599) print("The source file was", yaml_newfile_fullpath, "and exists =", os.path.isfile(yaml_newfile_fullpath))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`600) with open(os.path.join(localdir, 'techcrunch_text.yaml'), 'w') as f:`
Upgrade yaml module dblume authored 5 months ago	`601) yaml.dump(items, f, default_flow_style=None, width=120)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`602) with codecs.open(os.path.join(localdir, 'techcrunch_unicode.yaml'), 'w', 'utf-8') as f:`
Upgrade yaml module dblume authored 5 months ago	`603) yaml.dump(items, f, default_flow_style=None, encoding='utf-8', width=120)`
Original 2010-09-03 version David Blume authored 6 years ago	`604)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`605) make_feed_file(items)`
Original 2010-09-03 version David Blume authored 6 years ago	`606)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`607) make_index_html(items, weekend_stats, weekday_stats)`
Original 2010-09-03 version David Blume authored 6 years ago	`608) else:`
Upgrade to Python3 dblume authored 5 months ago	`609) print("No entries were added this time.")`
Original 2010-09-03 version David Blume authored 6 years ago	`610)`
Catch up to production agai... David Blume authored 6 years ago	`611) except Exception as e:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`612) exceptional_text = "An exception occurred: " + str(e.__class__) + " " + str(e)`
Upgrade to Python3 dblume authored 5 months ago	`613) print(exceptional_text, ' '.join(progress_text))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`614) traceback.print_exc(file=sys.stdout)`
Original 2010-09-03 version David Blume authored 6 years ago	`615) try:`
Replace hardcoded string wi... David Blume authored 6 years ago	`616) send_email('Exception thrown in ' + os.path.basename(__file__),`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`617) exceptional_text + "\n" + traceback.format_exc(),`
Replace hardcoded string wi... David Blume authored 6 years ago	`618) (smtp_creds.default_recipient,))`
Catch up to production agai... David Blume authored 6 years ago	`619) except Exception as e:`
Upgrade to Python3 dblume authored 5 months ago	`620) print("Could not send email to notify you of the exception. :(")`
Original 2010-09-03 version David Blume authored 6 years ago	`621) 622) message = sys.stdout.getvalue() 623) sys.stdout = old_stdout 624) sys.stderr = old_stderr 625) if not debug:`
Upgrade to Python3 dblume authored 5 months ago	`626) print(message)`
Original 2010-09-03 version David Blume authored 6 years ago	`627) 628) # Finally, let's save this to a statistics page`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`629) if os.path.exists(os.path.join(localdir, 'stats.txt')): 630) with open(os.path.join(localdir, 'stats.txt')) as f:`
Original 2010-09-03 version David Blume authored 6 years ago	`631) lines = f.readlines() 632) else: 633) lines = []`
Catch up to production agai... David Blume authored 6 years ago	`634) lines = lines[:672] # Just keep the past week's worth`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`635) # status = len(message.strip()) and message.strip().replace('\n', ' - ') or "OK" 636) status = len(message.strip()) and '\n '.join( message.splitlines()) or "OK" 637) lines.insert(0, "%s %3.0fs %s\n" % (time.strftime('%Y-%m-%d, %H:%M', time.localtime()), time.time() - start_time, status)) 638) with open(os.path.join(localdir,'stats.txt' ), 'w') as f: 639) f.writelines(lines)`