techcrunch.py (9179b4b) - techcrunch.git

9179b4b6a9f34c3d241400ca691a88c9b4ed30f9

Upgrade to Python3 dblume authored 5 months ago	`1) #!/home/dblume/opt/python-3.9.6/bin/python3`
Original 2010-09-03 version David Blume authored 6 years ago	`2) #`
Moved SMTP credentials to s... David Blume authored 6 years ago	`3) # Testing without affecting the yaml file and saving the updated one aside: 4) # cp techcrunch.yaml techcrunch.yaml_back; ./techcrunch.py; \ 5) # cp techcrunch.yaml techcrunch_tmp.yaml; cp techcrunch.yaml_back techcrunch.yaml`
Original 2010-09-03 version David Blume authored 6 years ago	`6) import feedparser 7) import sys`
Upgrade yaml module dblume authored 5 months ago	`8) if sys.path[0] != '': 9) sys.path.insert(0, '') # DXB Only needed for cronjobs to find yaml 10) import yaml`
Original 2010-09-03 version David Blume authored 6 years ago	`11) import os 12) import time 13) import codecs 14) import traceback 15) import calendar 16) import pickle`
Upgrade to Python3 dblume authored 5 months ago	`17) import urllib.request, urllib.parse, urllib.error 18) import http.client`
Original 2010-09-03 version David Blume authored 6 years ago	`19) import shutil 20) import smtplib 21) import analysis`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`22) import json`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`23) import xml`
2011-02-04: Update to the c... David Blume authored 6 years ago	`24) import operator`
Upgrade to Python3 dblume authored 5 months ago	`25) import html 26) import io`
Moved SMTP credentials to s... David Blume authored 6 years ago	`27) import smtp_creds # Your own credentials, used in send_email()`
Original 2010-09-03 version David Blume authored 6 years ago	`28) 29) debug = True 30) any_entry_added = False`
Google terminated image cha... David Blume authored 5 years ago	`31) tags_to_post = {'apple', 'google', 'roku'}`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`32) authors_to_post = ['michael arrington',] 33)`
Upgrade yaml module dblume authored 5 months ago	`34) # 2022-02-24: Seeing too many posts with just mean + sigma, adjust with factor 35) threshold_sigma_factor = 1.5 36)`
Catch up to production agai... David Blume authored 6 years ago	`37) # TODO 2018-01-18: Maybe combine fb_likes with bf_shares or something... 38) rhs_metric = 'fb_likes'`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`39) rhs_metric_times = 'comment_times'`
Original 2010-09-03 version David Blume authored 6 years ago	`40) 41) localdir = '' 42)`
Google terminated image cha... David Blume authored 5 years ago	`43) html_head = """<!DOCTYPE html> 44) <html><head>`
Original 2010-09-03 version David Blume authored 6 years ago	`45) <title>TechCrunch Feed Filter</title> 46) <!-- <link rel="alternate" type="application/rss+xml" title="RSS feed" href="http://techcrunch.dlma.com/rss_feed.xml" /> --> 47) <link rel="alternate" type="application/rss+xml" title="RSS feed" href="http://feeds.feedburner.com/TrendingAtTechcrunch" /> 48) <style type="text/css"> 49) body { font-family: "Arial", san-serif; }`
Fix comment, and update a URL. David Blume authored 5 years ago	`50) .author { font-size: smaller; color:gray; } 51) .date { font-size: smaller; color:gray; }`
Original 2010-09-03 version David Blume authored 6 years ago	52) .h3 { font-size: larger; } 53) a { text-decoration: none; } 54) /* table { border: none; border-collapse:collapse; font-size: large } */ 55) table { border-collapse: collapse; } 56) table.legend { border:1px solid LightSlateGray; font-size: medium; border-collapse:separated; } 57) table.legend th { border: 1px solid LightSlateGray; background-color: #E0E0E0; } 58) table.legend td { border: 1px solid LightSlateGray; } 59) tr.even { background:#%s; padding: 2em; } 60) tr.odd { background:#%s; padding-bottom: 2em; }
Little cleanup, div to rese... David Blume authored 5 years ago	`61) td div { height: 68px; }`
Original 2010-09-03 version David Blume authored 6 years ago	`62) </style>`
Google terminated image cha... David Blume authored 5 years ago	`63) <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script> 64) <script type="text/javascript"> 65) google.charts.load('current', {'packages':['corechart']}); 66) google.charts.setOnLoadCallback(drawChart); 67) function drawChart() { 68) var options = {`
Fix comment, and update a URL. David Blume authored 5 years ago	`69) width:%d,`
Google terminated image cha... David Blume authored 5 years ago	`70) height:68, 71) pointSize:0.1, 72) dataOpacity:1.0, 73) series: { 0: {targetAxisIndex:0}, 1: {targetAxisIndex:1, color:'limegreen'} }, 74) vAxis: { gridlines: {count: 0}, maxValue: 1 }, 75) hAxis: { gridlines: {count: 0}, ticks: [] }, 76) vAxes: { 0: {textStyle: {fontSize: 11, color: 'blue'} }, 1: {viewWindowMode: 'maximized', baselineColor: '#A0D0A0', textStyle: {fontSize: 11, color: 'limegreen'} } }, 77) }; 78) %s 79) } 80) </script> 81) </head> 82) <body>`
Original 2010-09-03 version David Blume authored 6 years ago	`83) <div align='center'><h3>TechCrunch Feed Filter</h3></div>`
Add feed icon and link to t... David Blume authored 6 years ago	`84) This page shows what analysis is done to filter the noise away from the Techcrunch feed into 85) <a href="http://feeds.feedburner.com/TrendingAtTechcrunch"> a more concise feed <img src="feed.png" alt="feed" height="14" width="14"></a>.`
Google terminated image cha... David Blume authored 5 years ago	`86) <a href="https://david.dlma.com/blog/my-techcrunch-feed-filter">Learn more about the Feed Filter</a>.<br /><br />`
Original 2010-09-03 version David Blume authored 6 years ago	`87) """ 88) 89) html_footer = """ 90) </table> 91) </div><br /> 92) <div align='center'>Thanks to <a href="http://www.feedparser.org/">The Universal Feed Parser module</a>,`
Add feed icon and link to t... David Blume authored 6 years ago	`93) <a href="http://pyyaml.org/">PyYAML</a> and <a href="http://code.google.com/apis/chart/">Google Charts</a>.<br />`
Google terminated image cha... David Blume authored 5 years ago	94) <a href="http://git.dlma.com/techcrunch.git/">source</a> • <a href="techcrunch.yaml">raw data</a> • <a href="stats.txt">status</a><br />© 2011 <a href="https://david.dlma.com">David Blume</a></div><br /> 95) </body> 96) </html> 97) """ 98) 99) chart_data_header = """ var data = google.visualization.arrayToDataTable([ 100) ['', 'Comments', 'Shares', {'type': 'string', 'role': 'style'}], 101) """ 102) chart_data_middle = """ ]); 103) var chart = new google.visualization.LineChart(document.getElementById('chart%d')); 104) options.backgroundColor = '#%s';
Original 2010-09-03 version David Blume authored 6 years ago	`105) """ 106)`
Fix comment, and update a URL. David Blume authored 5 years ago	`107) img_width = 240`
Google terminated image cha... David Blume authored 5 years ago	`108) img_height = 68`
Original 2010-09-03 version David Blume authored 6 years ago	`109) 110) series_1_color = "0000FF" 111) series_2_color = "00AA00" 112) threshold_color = "FF8C00"`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`113) tag_color = "F01000"`
Original 2010-09-03 version David Blume authored 6 years ago	`114) 115) even_background = "F8F8F8" 116) odd_background = "E8E8E8" 117)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`118) even_watermark = "E0E0FF" 119) odd_watermark = "D0D0F0" 120) 121)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`122) def asciiize(s):`
Original 2010-09-03 version David Blume authored 6 years ago	`123) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`124) return s.encode('ascii')`
Better conformance to PEP-8... David Blume authored 6 years ago	`125) except UnicodeEncodeError:`
Original 2010-09-03 version David Blume authored 6 years ago	`126) return s`
Better conformance to PEP-8... David Blume authored 6 years ago	`127) except exceptions.AttributeError:`
Original 2010-09-03 version David Blume authored 6 years ago	`128) return s 129)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`130)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`131) def send_email(subject, message, toaddrs, 132) fromaddr='"%s" <%s>' % (os.path.basename(__file__), smtp_creds.user)): 133) """ Sends Email """ 134) smtp = smtplib.SMTP(smtp_creds.server, port=smtp_creds.port) 135) smtp.login(smtp_creds.user, smtp_creds.passw)`
Better conformance to PEP-8... David Blume authored 6 years ago	`136) smtp.sendmail(fromaddr, 137) toaddrs,`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`138) "Content-Type: text/plain; charset=\"us-ascii\"\r\nFrom: %s\r\nTo: %s\r\nSubject: %s\r\n%s" % \ 139) (fromaddr, ", ".join(toaddrs), subject, message))`
Original 2010-09-03 version David Blume authored 6 years ago	`140) smtp.quit() 141)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`142)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`143) def index_id(a_list, op, elem):`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`144) try:`
Upgrade to Python3 dblume authored 5 months ago	`145) return next((index for index, item in enumerate(a_list) if op(item, elem)))`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`146) except: 147) return -1 148)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`149)`
Google terminated image cha... David Blume authored 5 years ago	`150) def write_chart_data(time_posted, lhs_times, lhs_values, rhs_times, 151) rhs_values, threshold_value, image_index, tag_hit, chart_io):`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`152) # lhs_times, lhs_values = zip(comments) 153) # rhs_times, rhs_values = zip(rhs)`
Google terminated image cha... David Blume authored 5 years ago	`154) is_odd_row = image_index % 2`
Original 2010-09-03 version David Blume authored 6 years ago	`155)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`156) if not len(lhs_times): 157) lhs_times = [time_posted,] 158) if not len(lhs_values): 159) lhs_values = [0,] 160) if not len(rhs_times): 161) rhs_times = [time_posted,] 162) if not len(rhs_values): 163) rhs_values = [0,] 164) 165) lhs_times = [(i - time_posted) / 1800 for i in lhs_times] 166) rhs_times = [(i - time_posted) / 1800 for i in rhs_times] 167)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`168) met_threshold_pt = -1 169) if threshold_value != -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`170) met_threshold_pt = index_id(rhs_values, operator.ge, threshold_value)`
2011-02-04: Update to the c... David Blume authored 6 years ago	`171) if met_threshold_pt == -1 or tag_hit: 172) # This can happen if threshold_value was set to a number 173) # because the author or a tag was matched, but the article 174) # was unpopular. We choose to put a marker at point index 0. 175) met_threshold_pt = 0 176) 177) if is_odd_row != 0: 178) bg_color = even_background 179) else: 180) bg_color = odd_background`
Google terminated image cha... David Blume authored 5 years ago	`181) 182) chart_io.write(chart_data_header) 183) for i in range(8): 184) if i == met_threshold_pt: 185) if tag_hit: 186) style = "'point { size: 5; fill-color: #FF0000; shape-type: diamond}'" 187) else: 188) style = "'point { size: 5; fill-color: #FF8C00; }'" 189) else: 190) style = "null" 191) if i < len(lhs_values): 192) lhs_value = str(lhs_values[i]) 193) else: 194) lhs_value = "null" 195) if i < len(rhs_values): 196) rhs_value = str(rhs_values[i])`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`197) else:`
Google terminated image cha... David Blume authored 5 years ago	`198) rhs_value = "null" 199) chart_io.write(" [%d, %s, %s, %s],\n" % (i, lhs_value, rhs_value, style)) 200) chart_io.write(chart_data_middle % (image_index, bg_color)) 201) if met_threshold_pt == -1 and not tag_hit: 202) chart_io.write(" delete options.vAxes[1].baseline;\n") 203) else: 204) chart_io.write(" options.vAxes[1].baseline = %d;\n" % (threshold_value,)) 205) chart_io.write(" chart.draw(data, options);\n\n")`
Original 2010-09-03 version David Blume authored 6 years ago	`206)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`207)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`208) def process_feed(yaml_items):`
Moved SMTP credentials to s... David Blume authored 6 years ago	`209) """Retrieve the url and process it.`
Original 2010-09-03 version David Blume authored 6 years ago	`210) feed_info (in, out) A tuple that describes an individual feed, like its name and etag. 211) """`
Upgrade yaml module dblume authored 5 months ago	`212) #feed = feedparser.parse('https://techcrunch.com/feed/') 213) feed = feedparser.parse('https://pi.dlma.com/techcrunch_feed.php') # DXB temporary until removed from denylist`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`214) if hasattr(feed, 'status'):`
Original 2010-09-03 version David Blume authored 6 years ago	`215) if feed.status == 304: 216) pass 217) else: 218) if feed.status != 200 and feed.status != 307 and feed.status != 301 and feed.status != 302: 219) if feed.status == 503:`
Upgrade to Python3 dblume authored 5 months ago	`220) print("the feed is temporarily unavailable.")`
Original 2010-09-03 version David Blume authored 6 years ago	`221) elif feed.status == 400:`
Upgrade to Python3 dblume authored 5 months ago	`222) print("the feed says we made a bad request.")`
Original 2010-09-03 version David Blume authored 6 years ago	`223) elif feed.status == 502:`
Upgrade to Python3 dblume authored 5 months ago	`224) print("the feed reported a bad gateway error.")`
Original 2010-09-03 version David Blume authored 6 years ago	`225) elif feed.status == 404:`
Upgrade to Python3 dblume authored 5 months ago	`226) print("the feed says the page was not found.")`
Original 2010-09-03 version David Blume authored 6 years ago	`227) elif feed.status == 500:`
Upgrade to Python3 dblume authored 5 months ago	`228) print("the feed had an internal server error.")`
Original 2010-09-03 version David Blume authored 6 years ago	`229) elif feed.status == 403:`
Upgrade to Python3 dblume authored 5 months ago	`230) print("Access to the feed was forbidden.")`
Original 2010-09-03 version David Blume authored 6 years ago	`231) else:`
Upgrade to Python3 dblume authored 5 months ago	`232) print("the feed returned feed.status %d." % ( feed.status, ))`
Original 2010-09-03 version David Blume authored 6 years ago	`233) else: 234) # Save off this`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`235) if hasattr(feed, 'bozo_exception') and isinstance(feed.bozo_exception, xml.sax._exceptions.SAXParseException):`
Upgrade to Python3 dblume authored 5 months ago	`236) print("Didn't pickle TechCrunch feed because it had a bozo_exception: %s" % (str(feed.bozo_exception)))`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`237) else: 238) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`239) with open(os.path.join(localdir, 'techcrunch_feed.pickle'), 'wb') as f: 240) pickle.dump(feed, f)`
Catch up to production agai... David Blume authored 6 years ago	`241) except(pickle.PicklingError, exceptions.TypeError) as e:`
Upgrade to Python3 dblume authored 5 months ago	`242) print("An error occurred while pickling the feed: %s." % \`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`243) (# str(e.__class__),`
Upgrade to Python3 dblume authored 5 months ago	`244) str(e)))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`245) traceback.print_exc(3, file=sys.stdout)`
Original 2010-09-03 version David Blume authored 6 years ago	`246)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`247) for i in reversed(feed.entries): 248) process_item(i, yaml_items)`
Original 2010-09-03 version David Blume authored 6 years ago	`249) 250) # If we have more than 200 items, remove the old ones.`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`251) while len(yaml_items) > 200:`
Original 2010-09-03 version David Blume authored 6 years ago	`252) yaml_items.pop() 253) 254) for i in yaml_items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`255) # i['title'] = asciiize(i['title']) 256) # i['tags'] = map(asciiize, i['tags']) 257) process_yaml_item(i)`
Original 2010-09-03 version David Blume authored 6 years ago	`258) 259) else: 260) if hasattr(feed, 'bozo_exception'): 261) e = feed.bozo_exception`
Upgrade to Python3 dblume authored 5 months ago	`262) if isinstance(e, urllib.error.URLError):`
Original 2010-09-03 version David Blume authored 6 years ago	`263) print_last_line = True 264) if hasattr(e, 'reason'): 265) if e.reason[0] == 110:`
Upgrade to Python3 dblume authored 5 months ago	`266) print("the feed's connection timed out.")`
Original 2010-09-03 version David Blume authored 6 years ago	`267) print_last_line = False 268) elif e.reason[0] == 111:`
Upgrade to Python3 dblume authored 5 months ago	`269) print("the feed's connection was refused.")`
Original 2010-09-03 version David Blume authored 6 years ago	`270) print_last_line = False 271) elif e.reason[0] == 104:`
Upgrade to Python3 dblume authored 5 months ago	`272) print("the feed reset the connection.")`
Original 2010-09-03 version David Blume authored 6 years ago	`273) print_last_line = False 274) else:`
Upgrade to Python3 dblume authored 5 months ago	`275) print("the feed had a URLError with reason %s." % (str(e.reason),))`
Original 2010-09-03 version David Blume authored 6 years ago	`276) print_last_line = False 277) if print_last_line:`
Upgrade to Python3 dblume authored 5 months ago	`278) print("the feed had a URLError %s" % (str(e),)) 279) elif isinstance(e, http.client.BadStatusLine): 280) print("the feed gave a bad status line. (%s)" % (str(e),))`
Original 2010-09-03 version David Blume authored 6 years ago	`281) else:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`282) if len(str(e)):`
Upgrade to Python3 dblume authored 5 months ago	`283) print("the feed bozo_exception: %s \"%s\"" % (str(e.__class__), str(e)))`
Original 2010-09-03 version David Blume authored 6 years ago	`284) else:`
Upgrade to Python3 dblume authored 5 months ago	`285) print("the feed bozo_exception: %s %s" % (str(e.__class__), repr(e)))`
Original 2010-09-03 version David Blume authored 6 years ago	`286) else:`
Upgrade to Python3 dblume authored 5 months ago	`287) print("the feed returned class %s, %s" % (str(feed.__class__), str(feed)))`
Original 2010-09-03 version David Blume authored 6 years ago	`288)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`289)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`290) def process_item(feed_item, yaml_items):`
Catch up to production agai... David Blume authored 6 years ago	`291) """Processes an RSS feed item, and converts it to a YAML item"""`
Original 2010-09-03 version David Blume authored 6 years ago	`292) # Get the time 293) global any_entry_added`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`294) timecode_now = int(time.time())`
Original 2010-09-03 version David Blume authored 6 years ago	`295) date_parsed = time.gmtime()`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`296) if hasattr(feed_item, 'issued_parsed'):`
Original 2010-09-03 version David Blume authored 6 years ago	`297) date_parsed = feed_item.issued_parsed`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`298) elif hasattr(feed_item, 'date_parsed'):`
Original 2010-09-03 version David Blume authored 6 years ago	`299) date_parsed = feed_item.date_parsed 300) else:`
Upgrade to Python3 dblume authored 5 months ago	`301) print("process_item found no timestamp for", asciiize(feed_item.link))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`302) timecode_parsed = calendar.timegm(date_parsed)`
Original 2010-09-03 version David Blume authored 6 years ago	`303)`
Catch up to production agai... David Blume authored 6 years ago	`304) link = feed_item.link 305) if hasattr(feed_item, 'feedburner_origlink'): 306) link = feed_item.feedburner_origlink 307) 308) # TODO 2018-01-18: Leave in the ncid for URL clicks, but remove during processing. 309) # suffix_to_remove = '?ncid=rss' 310) # if link.endswith(suffix_to_remove): 311) # link = link[:-len(suffix_to_remove)] 312)`
Original 2010-09-03 version David Blume authored 6 years ago	`313) # Look for i.feedburner_origlink in yaml_items 314) yaml_item = None 315) for i in yaml_items:`
Catch up to production agai... David Blume authored 6 years ago	`316) if link == i['link']:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`317) yaml_item = i 318) break`
Catch up to production agai... David Blume authored 6 years ago	`319) if yaml_item is None:`
Original 2010-09-03 version David Blume authored 6 years ago	`320) author = ''`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`321) if hasattr(feed_item, 'author'): 322) author = asciiize(feed_item.author)`
Original 2010-09-03 version David Blume authored 6 years ago	`323)`
Catch up to production agai... David Blume authored 6 years ago	`324) # Make a new yaml_item`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`325) yaml_item = {'title' : asciiize(feed_item.title), 326) 'link' : asciiize(link), 327) 'author' : author, 328) 'tags' : [], 329) 'orig_posted' : timecode_parsed, 330) 'qualified' : -1, 331) 'comment_times' : [], 332) 'fb_comments' : [], 333) 'fb_shares' : [], 334) 'fb_likes' : [], 335) 'slash_comment_times' : [], 336) 'slash_comments' : []`
Original 2010-09-03 version David Blume authored 6 years ago	`337) }`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`338) if hasattr(feed_item, 'tags'):`
Original 2010-09-03 version David Blume authored 6 years ago	`339) for i in feed_item.tags:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`340) yaml_item['tags'].append(asciiize(i.term))`
Original 2010-09-03 version David Blume authored 6 years ago	`341)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`342) yaml_items.insert(0, yaml_item)`
Original 2010-09-03 version David Blume authored 6 years ago	`343) any_entry_added = True 344) 345) # Maybe check to ensure that this item isn't too old. 346) if timecode_parsed < timecode_now - 60 * 30 * 9: 347) return 348) 349) # Now, add the new values`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`350) if hasattr(feed_item, 'slash_comments') and len(yaml_item['slash_comments']) < 8:`
Original 2010-09-03 version David Blume authored 6 years ago	`351) any_entry_added = True`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`352) yaml_item['slash_comment_times'].append(timecode_now) 353) yaml_item['slash_comments'].append(int(feed_item.slash_comments))`
Original 2010-09-03 version David Blume authored 6 years ago	`354)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`355)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`356) def process_yaml_item(yaml_item):`
Original 2010-09-03 version David Blume authored 6 years ago	`357) global any_entry_added 358)`
Catch up to production agai... David Blume authored 6 years ago	`359) # Related to TODO 2018-01-18: Remove ncid only during processing. 360) link = yaml_item['link'] 361) suffix_to_remove = '?ncid=rss' 362) # Maybe we should find() it instead, in case feedburner adds other options 363) if link.endswith(suffix_to_remove): 364) link = link[:-len(suffix_to_remove)] 365)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`366) timecode_now = int(time.time()) 367) if len(yaml_item['fb_comments']) < 8:`
Catch up to production agai... David Blume authored 6 years ago	`368) num_shares, num_comments, num_likes = Get_fb_stats(link)`
Original 2010-09-03 version David Blume authored 6 years ago	`369) if num_comments != -1: 370) any_entry_added = True`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`371) yaml_item['comment_times'].append(timecode_now) 372) yaml_item['fb_shares'].append(num_shares) 373) yaml_item['fb_comments'].append(num_comments) 374) yaml_item['fb_likes'].append(num_likes)`
Original 2010-09-03 version David Blume authored 6 years ago	`375)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`376) # if len(yaml_item['reddit_']) < 8:`
Better conformance to PEP-8... David Blume authored 6 years ago	`377) # num_ = get_reddit_stats(link)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`378) # if num_ != -1: 379) # any_entry_added = True 380) # yaml_item['reddit_times'].append(timecode_now) 381) # yaml_item['reddit_'].append(num_)`
Original 2010-09-03 version David Blume authored 6 years ago	`382)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`383)`
Better conformance to PEP-8... David Blume authored 6 years ago	`384) def get_reddit_stats(url_string):`
Catch up to production agai... David Blume authored 6 years ago	`385) """ Consider curl "https://www.reddit.com/api/info.json?url=http://i.imgur.com/HG9dJ.jpg"`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`386) """`
Original 2010-09-03 version David Blume authored 6 years ago	`387) return -1 388)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`389)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`390) def Get_fb_stats(url_string):`
Moved SMTP credentials to s... David Blume authored 6 years ago	`391) """Use graph's "engagement" field to get reactions and shares."""`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`392) shares = -1 393) comments = -1`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`394) likes = -1 395)`
Catch up to production agai... David Blume authored 6 years ago	`396) url_string = url_string.encode('utf-8') 397)`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`398) try:`
Upgrade to Python3 dblume authored 5 months ago	`399) encoded = urllib.parse.urlencode({'access_token': facebook_token})`
Catch up to production agai... David Blume authored 6 years ago	`400) url = 'https://graph.facebook.com/v2.11/?id=%s&fields=engagement&%s'`
Upgrade to Python3 dblume authored 5 months ago	`401) f = urllib.request.urlopen(url % (urllib.parse.quote_plus(url_string), encoded))`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`402) data = f.read() 403) f.close()`
Upgrade to Python3 dblume authored 5 months ago	`404) except (urllib.error.URLError, http.client.BadStatusLine) as e:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`405) if hasattr(e, 'reason'): # URLError`
Catch up to production agai... David Blume authored 6 years ago	`406) if hasattr(e, 'code'):`
Upgrade to Python3 dblume authored 5 months ago	`407) print("Get_fb_stats got an error (1):", e.code, e.reason, url_string)`
Catch up to production agai... David Blume authored 6 years ago	`408) else:`
Upgrade to Python3 dblume authored 5 months ago	`409) print("Get_fb_stats got an error (2):", e.reason, url_string)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`410) elif hasattr(e, 'code'): #URLError`
Upgrade to Python3 dblume authored 5 months ago	`411) print("Get_fb_stats got an error. Code:", e.code, url_string)`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`412) else:`
Upgrade to Python3 dblume authored 5 months ago	`413) print("Get_fb_stats got an error (3):", str(e))`
Catch up to production agai... David Blume authored 6 years ago	`414) return shares, comments, likes`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`415) if len(data) > 20:`
Catch up to production agai... David Blume authored 6 years ago	`416) d = json.loads(data)['engagement'] 417) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`418) shares = d['share_count']`
Catch up to production agai... David Blume authored 6 years ago	`419) except KeyError:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`420) shares = 0`
Catch up to production agai... David Blume authored 6 years ago	`421) 422) try: 423) likes = d['reaction_count'] 424) except KeyError: 425) likes = 0 426) 427) # TODO 2018-01-18: og_object metric was likes + shares + comments 428) # Here we'll combine likes and shares, and comments with plugin_comments 429) likes += shares 430) 431) try: 432) comments = d['comment_plugin_count'] + d['comment_count'] 433) except KeyError: 434) comments = 0`
2013-08-04: Miscellaneous c... David Blume authored 6 years ago	`435) else:`
Upgrade to Python3 dblume authored 5 months ago	`436) print("Get_fb_stats got too little data for ", url_string)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`437) return shares, comments, likes`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`438) 439)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`440) def make_index_html(yaml_items, weekend_stats, weekday_stats): 441) """Writes a static index.html file from the YAML items."""`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`442) cur_time = int(time.time()) 443) new_index_fullpath = os.path.join(localdir, 'index.html_new') 444) index_fullpath = os.path.join(localdir, 'index.html')`
Original 2010-09-03 version David Blume authored 6 years ago	`445)`
Upgrade to Python3 dblume authored 5 months ago	`446) chart_io = io.StringIO()`
Google terminated image cha... David Blume authored 5 years ago	`447) for image_index, image in enumerate(yaml_items[:40]): 448) tag_hit = False 449) if image['author'].lower() in authors_to_post: 450) tag_hit = True 451) elif len(set([j.lower() for j in image['tags']]) & tags_to_post) > 0: 452) tag_hit = True 453) write_chart_data(image['orig_posted'], 454) image['comment_times'], 455) image['fb_comments'], 456) image[rhs_metric_times], 457) image[rhs_metric], 458) image['qualified'], 459) image_index, 460) tag_hit, 461) chart_io 462) )`
Catch up to production agai... David Blume authored 6 years ago	`463)`
Google terminated image cha... David Blume authored 5 years ago	`464) with codecs.open(new_index_fullpath, 'w', 'utf-8') as f:`
Fix comment, and update a URL. David Blume authored 5 years ago	`465) f.write(html_head % (even_background, odd_background, img_width, chart_io.getvalue()))`
Google terminated image cha... David Blume authored 5 years ago	`466) chart_io.close()`
Upgrade yaml module dblume authored 5 months ago	467) f.write('<div align="center">\n<table class="legend">\n<tr><th></th><th>Median</th><th>Mean</th><th>Std. Dev</th><th>Threshold (mean + sigma * %1.1f)</th></tr>\n' % threshold_sigma_factor) 468) f.write('<tr><th>Weekday</th><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td></tr>\n' % (weekday_stats[2][0], weekday_stats[2][1], weekday_stats[2][2], weekday_stats[2][1] + weekday_stats[2][2] * threshold_sigma_factor)) 469) f.write('<tr><th>Weekend</th><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td><td>%1.1f</td></tr>\n' % (weekend_stats[2][0], weekend_stats[2][1], weekend_stats[2][2], weekend_stats[2][1] + weekend_stats[2][2] * threshold_sigma_factor))
Catch up to production agai... David Blume authored 6 years ago	`470) f.write('</table></div>\n<br />\n') 471) f.write('<div align="center">\n<table>\n') 472) for image_index, image in enumerate(yaml_items[:40]):`
Fix comment, and update a URL. David Blume authored 5 years ago	`473) f.write('<tr valign="center" class="%s">\n <td><strong><a href="%s">%s</a></strong> <span class="date">at %s</span> <span class="author">by %s</span></td>\n' % \`
Catch up to production agai... David Blume authored 6 years ago	`474) (image_index % 2 and "even" or "odd", 475) image['link'], 476) image['title'].encode('ascii', 'xmlcharrefreplace'),`
Fix comment, and update a URL. David Blume authored 5 years ago	`477) time.strftime("%H:%M", time.localtime(image['orig_posted'])).encode('ascii', 'xmlcharrefreplace'),`
Catch up to production agai... David Blume authored 6 years ago	`478) image['author'].encode('ascii', 'xmlcharrefreplace'), 479) ) 480) ) 481) f.write(' <td>%s<td>\n' % (image['qualified'] != -1 and '<img src="star_30.png" width="30" height="29" />' or ''))`
Google terminated image cha... David Blume authored 5 years ago	`482) f.write(' <td><div id="chart%d" /></td></tr>\n' % (image_index, ))`
Catch up to production agai... David Blume authored 6 years ago	`483) f.write(html_footer) 484)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`485) if os.path.exists(index_fullpath): 486) os.unlink(index_fullpath) 487) shutil.move(new_index_fullpath, index_fullpath)`
Original 2010-09-03 version David Blume authored 6 years ago	`488)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`489)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`490) def make_feed_file(yaml_items): 491) """Writes the RSS feed file with the YAML items."""`
Catch up to production agai... David Blume authored 6 years ago	`492) with codecs.open(os.path.join(localdir, 'rss_feed.xml'), 'wb', 'utf-8') as f:`
Fix comment, and update a URL. David Blume authored 5 years ago	`493) f.write("<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<rss version=\"2.0\" xmlns:atom=\"http://www.w3.org/2005/Atom\">\n<channel>\n<atom:link href=\"http://techcrunch.dlma.com/rss_feed.xml\" rel=\"self\" type=\"application/rss+xml\"/>\n<title>Trending at TechCrunch</title><link>http://techcrunch.dlma.com</link>")`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`494) f.write("<pubDate>%s</pubDate><description>Automatically Generated Feed</description><language>en-us</language>\n" % (time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`495) count = 0 496) for item in yaml_items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`497) now = time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime(item['orig_posted']))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`498) if item['qualified'] != -1:`
Upgrade to Python3 dblume authored 5 months ago	`499) escaped_title = html.escape(item['title']).encode('ascii', 'xmlcharrefreplace') 500) escaped_author = html.escape(item['author']).encode('ascii', 'xmlcharrefreplace')`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`501) f.write("<item><title>%s</title><pubDate>%s</pubDate><link>%s</link><guid isPermaLink=\"false\">%s</guid><description><![CDATA[By: %s]]></description></item>\n" % \ 502) (escaped_title, now, item['link'], item['link'], escaped_author))`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`503) count += 1 504) if count > 14: 505) break`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`506) f.write("</channel></rss>")`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`507)`
Original 2010-09-03 version David Blume authored 6 years ago	`508) 509) if __name__=='__main__': 510) start_time = time.time() 511) progress_text = [] 512) 513) old_stdout = sys.stdout 514) old_stderr = sys.stderr`
Upgrade to Python3 dblume authored 5 months ago	`515) sys.stdout = sys.stderr = io.StringIO()`
Original 2010-09-03 version David Blume authored 6 years ago	`516) 517) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`518) localdir = os.path.abspath(os.path.dirname(sys.argv[0]))`
Original 2010-09-03 version David Blume authored 6 years ago	`519) # 520) # Read in techcrunch.yaml 521) # 522) # [ { 'title' : 'Title Text', 523) # 'link' : u'http://techcrunch.com/2010/08/17/google-buzz-who-to-follow/', 524) # 'author' : u'MG Siegler', 525) # 'orig_posted' : 1282197199 526) # 'tags' : [ u'Google', u'privacy' ] 527) # 'qualified' : -1 528) # 'comment_times' : [ 1282197199, 1282197407 ]`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`529) # 'fb_comments' : [ 0, 5 ] 530) # 'fb_shares' : [ 0, 300 ] 531) # 'fb_likes' : [ 0, 19 ]`
Original 2010-09-03 version David Blume authored 6 years ago	`532) # 'slash_comment_times' : [ 1282197199, 1282197407 ] 533) # 'slash_comments' : [ 0, 5 ] 534) # }, 535) # { ... } 536) # ] 537) #`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`538) yaml_fullpath = os.path.join(localdir, 'techcrunch.yaml') 539) if os.path.exists(yaml_fullpath): 540) with open(yaml_fullpath, 'rb') as f:`
Upgrade yaml module dblume authored 5 months ago	`541) items = yaml.load(f, Loader=yaml.Loader)`
Catch up to production agai... David Blume authored 6 years ago	`542) if items is None:`
Upgrade to Python3 dblume authored 5 months ago	`543) print(yaml_fullpath, "exists, but was empty.")`
Catch up to production agai... David Blume authored 6 years ago	`544) items = []`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`545) 546) # Do any dictionary item updating that might be necessary 547) # for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`548) # if not item.has_key('fb_shares'):`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`549) # item['fb_shares'] = []`
Original 2010-09-03 version David Blume authored 6 years ago	`550) else:`
Upgrade to Python3 dblume authored 5 months ago	`551) print("could not open", yaml_fullpath)`
Original 2010-09-03 version David Blume authored 6 years ago	`552) items = [] 553)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`554) with open(os.path.join(localdir, 'facebook-token.txt'), 'r') as f:`
Upgrade to Python3 dblume authored 5 months ago	`555) json_obj = json.load(f) 556) facebook_token = json_obj['access_token']`
Catch up to production agai... David Blume authored 6 years ago	`557)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`558) progress_text = ["read techcrunch.yaml"] 559) process_feed(items)`
Original 2010-09-03 version David Blume authored 6 years ago	`560) 561) # 562) # If any work was done, then write files. 563) #`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`564) if any_entry_added:`
Better conformance to PEP-8... David Blume authored 6 years ago	`565) weekend_stats, weekday_stats = analysis.process_feed(items, rhs_metric, rhs_metric_times)`
Original 2010-09-03 version David Blume authored 6 years ago	`566)`
Fix comment, and update a URL. David Blume authored 5 years ago	`567) # We'll only look at the stats up to 2 hours after posting.`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`568) weekend_median, weekend_mean, weekend_sigma = weekend_stats[2]`
Upgrade yaml module dblume authored 5 months ago	`569) weekend_threshold = weekend_mean + weekend_sigma * threshold_sigma_factor`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`570) weekday_median, weekday_mean, weekday_sigma = weekday_stats[2]`
Upgrade yaml module dblume authored 5 months ago	`571) weekday_threshold = weekday_mean + weekday_sigma * threshold_sigma_factor`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`572) for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`573) wday = time.localtime(item['orig_posted']).tm_wday`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`574) if wday == 5 or wday == 6: 575) threshold = weekend_threshold 576) else: 577) threshold = weekday_threshold 578) if item['qualified'] == -1:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`579) for i in range(len(item[rhs_metric_times])): 580) r_time = item[rhs_metric_times][i]`
Fix comment, and update a URL. David Blume authored 5 years ago	`581) if r_time - item['orig_posted'] < 7200:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`582) if item[rhs_metric][i] >= threshold:`
New feed URL, added comment... dblume authored 2 years ago	`583) # Comment out when graph.facebook.com engagement returns only 0s.`
2011-02-04: Update to the c... David Blume authored 6 years ago	`584) item['qualified'] = threshold`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`585) break`
Fix comment, and update a URL. David Blume authored 5 years ago	`586) else: 587) break`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`588) 589) # Automatically add those items whose authors and tags I like 590) for item in items:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`591) if item['qualified'] == -1 and len(item[rhs_metric_times]) > 0:`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`592) if item['author'].lower() in authors_to_post:`
2011-02-04: Update to the c... David Blume authored 6 years ago	`593) item['qualified'] = threshold`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`594) elif len(set([j.lower() for j in item['tags']]) & tags_to_post) > 0:`
2011-02-04: Update to the c... David Blume authored 6 years ago	`595) item['qualified'] = threshold`
Original 2010-09-03 version David Blume authored 6 years ago	`596) 597) # 598) # Write out the updated yaml file. 599) #`
2011-02-04: Algorithm chang... David Blume authored 6 years ago	`600) 601) # For the one file we really use, write to a file on the side, then move it.`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`602) yaml_newfile_fullpath = os.path.join(localdir, 'techcrunch_temp_writable.yaml') 603) with open(yaml_newfile_fullpath, 'wb') as f:`
Upgrade yaml module dblume authored 5 months ago	`604) yaml.dump(items, f, default_flow_style=None, width=120)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`605) try:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`606) os.rename(yaml_newfile_fullpath, yaml_fullpath)`
2015-11-23: Resync svn with... David Blume authored 6 years ago	`607) except OSError as e:`
Upgrade to Python3 dblume authored 5 months ago	`608) print("The source file was", yaml_newfile_fullpath, "and exists =", os.path.isfile(yaml_newfile_fullpath))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`609) with open(os.path.join(localdir, 'techcrunch_text.yaml'), 'w') as f:`
Upgrade yaml module dblume authored 5 months ago	`610) yaml.dump(items, f, default_flow_style=None, width=120)`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`611) with codecs.open(os.path.join(localdir, 'techcrunch_unicode.yaml'), 'w', 'utf-8') as f:`
Upgrade yaml module dblume authored 5 months ago	`612) yaml.dump(items, f, default_flow_style=None, encoding='utf-8', width=120)`
Original 2010-09-03 version David Blume authored 6 years ago	`613)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`614) make_feed_file(items)`
Original 2010-09-03 version David Blume authored 6 years ago	`615)`
Moved SMTP credentials to s... David Blume authored 6 years ago	`616) make_index_html(items, weekend_stats, weekday_stats)`
Original 2010-09-03 version David Blume authored 6 years ago	`617) else:`
Upgrade to Python3 dblume authored 5 months ago	`618) print("No entries were added this time.")`
Original 2010-09-03 version David Blume authored 6 years ago	`619)`
Catch up to production agai... David Blume authored 6 years ago	`620) except Exception as e:`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`621) exceptional_text = "An exception occurred: " + str(e.__class__) + " " + str(e)`
Upgrade to Python3 dblume authored 5 months ago	`622) print(exceptional_text, ' '.join(progress_text))`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`623) traceback.print_exc(file=sys.stdout)`
Original 2010-09-03 version David Blume authored 6 years ago	`624) try:`
Replace hardcoded string wi... David Blume authored 6 years ago	`625) send_email('Exception thrown in ' + os.path.basename(__file__),`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`626) exceptional_text + "\n" + traceback.format_exc(),`
Replace hardcoded string wi... David Blume authored 6 years ago	`627) (smtp_creds.default_recipient,))`
Catch up to production agai... David Blume authored 6 years ago	`628) except Exception as e:`
Upgrade to Python3 dblume authored 5 months ago	`629) print("Could not send email to notify you of the exception. :(")`
Original 2010-09-03 version David Blume authored 6 years ago	`630) 631) message = sys.stdout.getvalue() 632) sys.stdout = old_stdout 633) sys.stderr = old_stderr 634) if not debug:`
Upgrade to Python3 dblume authored 5 months ago	`635) print(message)`
Original 2010-09-03 version David Blume authored 6 years ago	`636) 637) # Finally, let's save this to a statistics page`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`638) if os.path.exists(os.path.join(localdir, 'stats.txt')): 639) with open(os.path.join(localdir, 'stats.txt')) as f:`
Original 2010-09-03 version David Blume authored 6 years ago	`640) lines = f.readlines() 641) else: 642) lines = []`
Catch up to production agai... David Blume authored 6 years ago	`643) lines = lines[:672] # Just keep the past week's worth`
2015-11-27: Remove obsolete... David Blume authored 6 years ago	`644) # status = len(message.strip()) and message.strip().replace('\n', ' - ') or "OK" 645) status = len(message.strip()) and '\n '.join( message.splitlines()) or "OK" 646) lines.insert(0, "%s %3.0fs %s\n" % (time.strftime('%Y-%m-%d, %H:%M', time.localtime()), time.time() - start_time, status)) 647) with open(os.path.join(localdir,'stats.txt' ), 'w') as f: 648) f.writelines(lines)`