#biocworkshops

2019-05-03

Sean Davis (11:19:51): > @Sean Davis has joined the channel

Sean Davis (11:19:51): > set the channel description: Discuss proposal for how to implement bioc workshop book building each year

Levi Waldron (11:19:51): > @Levi Waldron has joined the channel

Marcel Ramos Pérez (11:19:51): > @Marcel Ramos Pérez has joined the channel

Lorena Pantano (11:19:51): > @Lorena Pantano has joined the channel

Sean Davis (11:20:11): > Hi,@Lorena Pantano@Marcel Ramos Pérez@Levi Waldron.

Sean Davis (11:20:43): > I wrote up a small proposal outlining a “vision” for how to build BiocWorkshops this year.https://docs.google.com/document/d/1Hycl88aAl7CARrYhmPz8zXYggwgIRsX9iN5PD8YtdOA/edit?usp=sharing - File (Google Docs): BiocWorkshopBuildsProposal

Sean Davis (11:22:34): > Levi and I wanted to open discussion on the topic and see if we could flesh out something that works for us.@Marcel Ramos Pérezand@Lorena Pantano, I was really interested to hear if you had a little time to work on details of the implementation.

Sean Davis (11:23:21): > And, of course, any other ideas or feedback on the quick proposal is welcome before we try to proceed.

Sean Davis (11:37:05): > And a quick test workspace:https://github.com/seandavi/BiocWorkshopPackageTest

Sean Davis (11:43:18): > I should add you all as committers on the test workspace. The piece that needs to be worked out yet is the getting vignette/bookdown piece.

Sean Davis (11:45:49): > We still need to establish: > 1. A mechanism for workshop authors to tell us which vignette to use (a simple csv file?) > 2. A mechanism to copy the .Rmd files from the installed packages. > 3. A mechanism to update _bookdown.yml as needed (could be done by hand, even) > 4. A mechanism to actually build the book (probably just bookdown or a simple wrapper).

Lorena Pantano (13:32:22): > thanks, looking into this, it is useful

Sean Davis (14:14:21): > I added simple vignette copying functionality.https://github.com/seandavi/BiocWorkshopPackageTest/commit/418515117b0213e8c8d6f191613b2429ade4c820

2019-05-05

Sean Davis (15:16:18): > <!channel>, I tried implementing the details above. Some issues come up: > 1. For packages coming from github, one needs to install and force vignette building to get the Rmd file included in the installed package. > 2. Only the Rmd, R, and html files are installed into the doc system.file location, not bibliography, imgs, etc. Also, even when trying to include markdown usingkeep_md, the markdown is not included in the installed package. > This leads me to think that we will likely have to get access to the actual vignette directory and build things there before copying needed materials over to the root WorkshopBook directory.

Sean Davis (15:16:54): > Testing code: > > BiocManager::install('seandavi/Bioc2019-iSEE-workshop',build_opts = c("--no-resave-data", "--no-manual"), build_vignettes = TRUE, force=TRUE, keep_md=TRUE) >

Levi Waldron (15:36:16): > @Sean Davisyou’re awesome.

2019-05-06

Lorena Pantano (15:06:44): > we can clone, go to vignette, build there with rmarkdown and copy all to the right place?

Lorena Pantano (15:07:12): > are we going to have a meeting to discuss this? It would be helpful to me

Sean Davis (15:41:06): > That sounds like a good plan,@Lorena Pantano.

Sean Davis (15:41:47): > @Marcel Ramos Pérez,@Lorena Pantano,@Levi Waldrondo you have any availability for a call this week?

Lorena Pantano (15:44:19): > I think@Aedin Culhanewanted to get involved? and we filled a doodle for a meeting, but maybe was not for the infrastructure part. A little confused about who is leading what part (I apologize in advance).

Aedin Culhane (15:44:24): > @Aedin Culhane has joined the channel

Sean Davis (15:47:05): > If there is a meeting set, perhaps we can use a bit of that time to discuss infrastructure.

2019-05-07

Levi Waldron (08:58:33): > @Aedin Culhanesent a doodle poll last week but I don’t think announced the time. The most popular time was tomorrow noon, although I now booked that.

Levi Waldron (08:59:47): > I can miss a meeting though, don’t need to work around me if@Sean Daviscan make it

Aedin Culhane (09:00:03): > Yes it was wed noon. Levi I asked about what phone/video conference software we were using for the meeting and I didn’t know if you or I was setting that up. Sorry for confusion

Aedin Culhane (09:00:56): > In the doodle poll. Wednesday noon worked for everyone

Levi Waldron (09:00:57): > I say just use Hangouts, only Martin can set up bluejeans

Levi Waldron (09:02:40): > Will you announce it to the Google group? Best to stick with it probably.

Sean Davis (09:23:36): > Wednesday noon works for me.

Sean Davis (09:25:06): > I’d also vote for hangouts, but if there is a need for something else, I have a webex account we can use.

Lorena Pantano (09:56:07): > :+1:

Aedin Culhane (12:20:14): > @Sean DavisI haven’t sent up a google hangout call link in advance (normally I just do call and invite others on at that time).

Aedin Culhane (12:20:32): > So I want sure how to send out the invite.

2019-05-08

Sean Davis (07:51:52): > <!channel>Everyone should have gotten an invite for the call today that includes a hangouts line. If I got an email wrong and you didn’t get one, just ping me with the correct email and I’ll resend.

Sean Davis (10:04:37): > added an integration to this channel: github

Sean Davis (10:05:19): > removed an integration from this channel: github

Sean Davis (11:59:51): > https://github.com/seandavi/BookdownUtils

Sean Davis (16:15:34): > Moved tohttps://github.com/seandavi/CollaborativeBookdown.

2019-05-15

Lorena Pantano (11:41:17): > just to confirm that if the vignettes refer to images withsystem.file(package='package', 'vignettes')when including images, then the book render fine

Lorena Pantano (11:41:58): > but for that authors need to install the their packages first to actually render their vignette…

Lorena Pantano (11:42:34): > knitr::include_graphics(file.path(system.file(package='dummychapter1', 'vignettes'), "index.png"))this works

Lorena Pantano (11:56:12): > it works as well for bibliography the same way

Levi Waldron (16:38:49): > Oh that’s great news!

Levi Waldron (16:40:14): > Do they also render properly inbrowseVignettes()andvignette()? I would think so, but just checking.

Marcel Ramos Pérez (18:01:50): > BTW I’ve added a template files and instructions for workshop forks. I’m still working on getting Travis updated to the latest release of Bioconductor

Lorena Pantano (21:57:14): > I think it works, Levi. just need to do:build_vignettes = TRUEwhen installing from local

2019-05-16

Levi Waldron (14:05:04): > You guys are awesome.

2019-05-21

Sean Davis (11:55:56): > So, looks like the plan then is to ask workshop authors to: > 1. Package everything for each workshop as an R package. > 2. Include a standard vignette as an Rmd in the vignettes directory (as for any package). > 3. Rather than including extra files like images, etc., in the vignettes directory, instead put them ininst/vignettesand reference them in the Rmd usingsystem.file. > 4. Number 3 requires that the package be “installed” before building the vignette or usingbuild_vignettes=TRUEwhen building the package.

Lorena Pantano (12:01:33): > yes, i think that should work. I can push to a github my dummy package and you can try to do a dummy bookdown?

Sean Davis (12:06:54): > That would be AWESOME! It would be useful to have two packages–one for the workshop itself, and a second package that installs the workshop package and does what is necessary to build the bookdown.

Sean Davis (12:09:53): > In essence, the “master” package just needs a DESCRIPTION file with workshop packages and remotes listed, bookdown yaml files, and perhaps a travis yaml file. I guess it would also need a quick function to copy appropriate RMD files from the workshop packages as well.

Levi Waldron (14:57:05): > Sounds great. So I think there is no need to ask for specially namedbibliography.bibfiles etc, either. Lorena, as soon as your dummy package is available, I’ll send a follow-up email to presenters asking that they useinst/vignettesandsystem.file, and provide your dummy package as an example.

2019-05-22

Kevin Rue-Albrecht (03:15:29): > @Kevin Rue-Albrecht has joined the channel

Lorena Pantano (07:41:24): > https://github.com/lpantano/dummychapter1

Lorena Pantano (07:41:59): > actually I linked the fullvignettesfolder toinst, although probably is ok to link only the files

Charlotte Soneson (10:54:37): > @Charlotte Soneson has joined the channel

2019-05-23

Marcel Ramos Pérez (11:42:31) (in thread): > I can add some of these instructions to theREADME.mdfile on the repo

Marcel Ramos Pérez (11:52:26) (in thread): > https://github.com/Bioconductor/BiocWorkshops2019/blob/master/README.md

Lorena Pantano (12:56:10): > @Sean Daviscan the Rmd that will be the chapters have yaml headers?

Lorena Pantano (13:00:44): > And@Levi Waldroncan I have an example of a syllabus?

Marcel Ramos Pérez (13:10:52): > I don’t think they can have full yaml headers. You can refer tohttps://github.com/Bioconductor/BiocWorkshops

Marcel Ramos Pérez (13:11:15): > I’m working on a small function to pull the workshops from aDESCRIPTIONfile

Levi Waldron (13:15:19): > https://github.com/Bioconductor/BioC2019/blob/master/docs/workshop-syllabus.md

Lorena Pantano (13:29:00): > cool, I added that as the vignette and reduce the header to the min

Lorena Pantano (13:30:26): > travis is falling but the issue is previous installation of the package, some error with the travis image I think

2019-05-24

Marcel Ramos Pérez (13:45:40): > I’ve added a few helper functions inhttps://github.com/Bioconductor/workshopbuilderI’m still working on some of the details namely moving all vignettes after package installations.

Marcel Ramos Pérez (13:46:53) (in thread): > I think this is an issue withBiocManager:disappointed:We’re still working on getting 1.30.6 uploaded

Sean Davis (14:01:19): > Looks great,@Marcel Ramos Pérez

Sean Davis (14:01:26): > Thx!

2019-05-27

Levi Waldron (08:35:02): > Thanks for all the work you’ve already done on the repo, its README.md, and chasing workshop authors already@Marcel Ramos Pérez! I was just sitting down to try to do some of this and realized you’ve already done most of it:slightly_smiling_face:

Levi Waldron (08:35:23): > Very greatly appreciated

Levi Waldron (22:10:30): > @Sean Davisare the old instructions “Update the DESCRIPTION file adding packages utilized in your workshop to theImportsfield.” still necessary in the individual packages? This was there last year because we had a single DESCRIPTION file and didn’t want hundreds of files in the DEPENDS. If we’re going to end up collating DESCRIPTION files into a single mega-DESCRIPTION then I think this directive still holds and should be clarified, but otherwise then I guess authors can decide between DEPENDS and IMPORTS as they wish.

2019-05-28

Sean Davis (06:39:50): > @Levi WaldronThe only thing that needs to be added to the description is the workshop package and its associated remote. Installing the workshop package itself will pick up dependencies for that workshop. Does that make sense?

Levi Waldron (06:41:53): > But will all the individual workshop’s DEPENDS end up getting loaded during the book building?

Levi Waldron (06:43:17): > Or will the chapters somehow get built individually, then collated?

Levi Waldron (06:44:11): > If the former, then people need to limit their use of DEPENDS. If the latter, it doesn’t matter.

Sean Davis (13:51:16): > So, small hiccup. Packages must be “built” before the .Rmd is present in the installed package tree.

Sean Davis (14:42:47): > @Lori Shepherd, I am trying to install some packages and am getting: a 404 error from:https://bioconductor.org/packages/3.10/data/annotation/src/contrib/PACKAGES

Lori Shepherd (14:42:51): > @Lori Shepherd has joined the channel

Sean Davis (14:46:22): > BiocManager::install(…., version="3.10")is the code that triggers thewarningwhen installing a bioconductor package, but anerrorwhen installing a github package.

Lori Shepherd (15:28:51): > We just Branch the annotations for devel about an hour ago the air should go away by sometime this evening

Lori Shepherd (15:29:23): > *the error should go away by this evening once they finish progating

Sean Davis (16:03:44): > Perfect! I figured as much, but I didn’t want to assume.

Levi Waldron (17:41:41) (in thread): > @Sean Davisdo you have thoughts about this question?

Levi Waldron (17:42:59): > I’ll clear up this confusion - my understanding is we don’t want any PRs toBioconductor/BiocWorkshops2019, and we’re not going to add all package requirements to the masterDESCRIPTIONfile…https://github.com/Bioconductor/BiocWorkshops2019/pull/19/files

Marcel Ramos Pérez (17:56:27): > Yes, we don’t need to use PRs. Thanks for making those changes in the instructions Levi. The only requirements we’re going to add to the masterDESCRIPTIONfile are the remotes that point to the workshop repositories

2019-05-30

Sean Davis (18:40:18): > Hi,@Marcel Ramos Pérez(and others who might be interested). I am still working a bit onhttps://github.com/seandavi/CollaborativeBookdown.

Sean Davis (18:44:28): > There are a few helper functions to do things like find installed vignettes, test package installs (since I want to be able to create a report), test vignette builds (apart from the workshop book). The vignette has a little detail. Finally, in DEVNOTES.md is small shell command to pull out URLs from the comments of open issues. In the future, we may want to generate the issues in a more automated and controlled fashion, but that is an aside.

FeiZhao (18:53:36): > @FeiZhao has joined the channel

Marcel Ramos Pérez (19:24:22): > Hi Sean,@Sean DavisPerhaps we can combine these efforts into a single package? I have done some of what you list here inhttps://github.com/Bioconductor/workshopbuilderI’ll be happy to move stuff over

Sean Davis (19:31:06): > I thought the same thing.

Sean Davis (19:35:01): > If it is OK with you, I’ll work a bit more tonight and tomorrow. Just a quick look, I see what you mean that we do have some of the same functionality. I leaned on theremotespackage to do some of the munging–that may not be the best plan for the long term, so….

2019-06-04

Sean Davis (16:49:34): > Hi,@Marcel Ramos Pérez. I have gotten sidetracked a bit. We should probably just go ahead with what you have. As an intermediate step, it might be helpful to have a “simple” version that simply installs packages and doesn’t necessarily build the bookdown. As a reminder, I used Packer last year to build an AMI.https://seandavi.github.io/2018/07/infrastructure-as-code-building-the-bioconductor-conference-ami-with-packer/ - Attachment (seandavi(s12)): Infrastructure-as-Code: Building the Bioconductor Conference AMI With Packer | seandavi(s12) > One of the main features of the annual Bioconductor Conference is the proportion of time spent working with code in the form of workshops. To support these workshops, we ask workshop presenters to supply Rmarkdown materials which we collate into workshop materials. Using literate programming approaches like Rmarkdown ensures that the workflows are self-consistent and work as expected. In addition to the Rmarkdown workshop materials, we also need a consistent computing environment that can support reasonably large computation, provide high-performance network and file system access, and is essentially unlimited in scale (we expect to have >150 participants, each with his/her own machine).

Sean Davis (16:50:45): > The script I used is available on github:https://github.com/seandavi/terraform-can/tree/master/packer

2019-06-10

Sean Davis (09:55:41): > Hi,@Marcel Ramos Pérez.@Martin Morganbrings up a good point on email–we should prioritize the workshop AMI over the gitbook. Would you be able to put together a workshop master package with the respective workshop packages in a DESCRIPTION file and the appropriate Remotes? With that, I can begin to work on the AMI.

Marcel Ramos Pérez (10:03:36): > Yes, I have a function for that. I’ll run it and push the changes to Bioconductor/BiocWorkshops2019

Sean Davis (10:05:11): > Awesome!

Sean Davis (10:05:31): > I’ll attempt a first pass at building the AMI this afternoon.

Marcel Ramos Pérez (12:55:31): > Hi@Sean DavisI’ve updated theDESCRIPTIONfile with the remotes. I got some warnings about package names not being ‘valid’ from thedescpackage. I don’t know if this affects their building and loading…

Kevin Rue-Albrecht (15:40:38): > Happy to rename the GitHub repo if it helps (kevinrue/Bioc2019-iSEE-workshop).

Kevin Rue-Albrecht (15:49:57): > Done (https://github.com/Bioconductor/BiocWorkshops2019/issues/2)

Sean Davis (20:20:05): > The first pass at the conference AMI is done. The AMI id isami-01fbb4898c7d353d7.

Sean Davis (20:26:04): > Building the AMI (from scratch) takes just about 1 hour on a 32 core machine.

2019-06-11

Sean Davis (06:26:09): > New AMI that contains all workshop packages:ami-0410ec351c317fdef

Sean Davis (07:56:06): > Could workshop authors please check that ALL packages needed in their vignettes are included as dependencies? These will show up asNOTEs in the check report, but the package will still fail to install when rebuilding the vignette. Seehttps://github.com/Bioconductor/BiocWorkshops2019/issues/32for what this looks like in practice.

Sean Davis (13:02:01): > Just making notes to those here. To install a package from github (including the workshop master package), including vignette building, something like this seems to do the trick: > > BiocManager::install('seandavi/GEOquery', update=TRUE, upgrade='always', build_opts='', build=TRUE, ask=FALSE) >

Sean Davis (13:03:18): > So, doing: > > BiocManager::install('Bioconductor/BiocWorkshops2019', update=TRUE, upgrade='always', build_opts='', build=TRUE, ask=FALSE) > > should get us an install of all dependent packages and builds of all the vignettes (which serves as a high-level test).

Sean Davis (13:06:14): > There are some problems with dependency specification in several of the packages (missing suggests, basically), so the above for the BiocWorkshops2019 repo will fail. I have been running the same command on individual packages for testing purposes.

Sean Davis (13:17:38): > @Marcel Ramos Pérezand@Lorena Pantano, are you doing individual package testing and, if so, how?

Marcel Ramos Pérez (13:34:27): > I’m not. I think that the original intent was to have workshop contributors set up Travis for individual testing.

Sean Davis (13:40:04): > Thanks.

Lorena Pantano (13:50:59): > not me either. i went through to check if they were using system.file and ask for that

Lorena Pantano (13:51:20): > we can ask to paste the output of check()

Lorena Pantano (13:51:36): > or if travis is working again then travis

Marcel Ramos Pérez (13:52:36): > Thanks Lorena for checking all the workshops

2019-06-12

Sean Davis (08:59:48): > All the workshop packages are installing at this point based on@Marcel Ramos Pérez’s list in the BiocWorkshops2019 DESCRIPTION file. Note that I had to change some of the remotes and package names, so the entirely automated process got us about 80% of the way to getting the repos right.

Sean Davis (09:01:48): > What we need to start doing now is to build the vignettes (see above). Unfortunately, this process is probably best done one package at a time since we need to communicate errors back to package authors, all of which so far in my checking have had to do with missing Suggests. These are transparent to the workshop authors unless they carefully check NOTES in check results or they install in a build environment that does not already include the necessary packages.

Sean Davis (09:03:19): > Once the vignettes are all building, we can concentrate on building the book.

Sean Davis (09:06:18): > Does anyone know if we could leverage the Bioc single package builder for doing the build and check of individual workshop packages? Alternatively, does anyone know if the code/instrumentation for the single package builder is portable?

Lori Shepherd (09:13:37): > The live SPB no - its too integrated with the issue_tracker - the code/instrumentation is documented at:https://github.com/Bioconductor/packagebuilder/blob/master/documentation/Developing.mdfor creating a local version

Lori Shepherd (09:19:48): > the local version doesn’t have the integration for building upon version bump (I dont think) it would only have the capability of running jobs manually -

2019-06-17

Lori Shepherd (10:52:37): > Let me know when there is an updated AMI with the packages built on them for the vignettes displaying - I’d like to try to do a test courses and have people test some launched instances from there to make sure all of that will work - ideally tomorrow or wed so I could potentially have the actually conference one on thur (fri morning at the latest)

Sean Davis (12:26:37): > The last day to submit changes to vignettes is today and several vignettes still have minor issues. I’ll get this done as soon as we get pretty close.

Sean Davis (12:29:12): > Does anyone have strong feelings about running inside docker (on the AMI) versus running directly on an AMI? The advantages of docker are obvious–easier testing, portability (no AMI needed for others to use), reproducibility. That said, docker-in-docker will not be set up (too fragile for me to get right), so I’ll have to make a backup plan for docker-in-workflow users (there are 2).

Lori Shepherd (12:51:25): > dont know how the running inside docker will play with the launched instances but other than that no strong feelings….

Sean Davis (15:26:58): > @Lori Shepherdwould it be possible to open ports 80 and 8787 on the AMIs? If we can do a separate security group for this year, that would be great.

Sean Davis (16:24:21): > The current status of packages is here:https://docs.google.com/spreadsheets/d/19w84Zve7Ria4wRuZ2G8HxBPhDu4fk6aMRTbqnBrjRAQ/edit?usp=sharing - File (Google Sheets): bioc2019_workshop_status

Kevin Rue-Albrecht (16:27:55): > I’m in the green/OK, but how do I know if I’m on the Google group email. Should have I received an email by now? I can’t see any.

Lori Shepherd (16:29:49) (in thread): > I’ll look into figuring out how to do this tomorrow morning

Sean Davis (16:30:09): > Sorry–ignore my comment about the google group.

Sean Davis (16:30:40) (in thread): > It will likely be in the launch template.

Sean Davis (16:34:28) (in thread): > The process would be to create a new security group (call it Bioc2019 if you like) that includes the ports and all IP addresses. Then, replace the old security group in the launch template with the Bioc2019 one.

2019-06-18

Lori Shepherd (07:51:36) (in thread): > 22 (for ssh) and 80 (http for rstudio) have always been open ….as well as 443 (https) … The current courses security group also has 8080 open which I think we original did in case of support for jupyter notebooks (as the descriptive labeling is apache serving jupyter) …. i dont know how ports work, would 8080 work or does it have to be 8787 - just wondering if creation of a new security group is really necessary or if we can utilize the one we have been using for courses/conferences

Sean Davis (10:46:11) (in thread): > 8080 is fine. Thanks. I am going to run the main course on 80 (which will be docker) and a second Rstudio on the AMI on port 8080 for the one workshop that needs direct access to docker. I’ll have an AMI to test this afternoon.

Lori Shepherd (10:51:20) (in thread): > great than I think we can use the same security group we’ve used in the past. We can test it once you have it set up. I’ll create a mock course using your AMI when its done and make sure it launches through the course helper.

Lori Shepherd (18:24:21) (in thread): > Any update? We haven’t had an AMI configured like this and I really want to test it with our course helper to make sure it works correctly. I didn’t create the helper so the sooner we can test the better….

Sean Davis (23:18:35) (in thread): > Hi,@Lori Shepherd. Sorry to take so long. I have been working with the last two authors to get things worked out with their workshop vignettes, but we still haven’t gotten those two to build successfully. The two problematic workshops are the CWL and the OSCA workshops.

Sean Davis (23:19:02) (in thread): > The AMI to move ahead with is:ami-0cb2d00d5cfab6d3f.

Sean Davis (23:24:22) (in thread): > The AMI has all workshops running inside Docker except the CWL workshop (since Docker-inside-docker is a hack). When the AMI starts up, the Docker Rstudio is on Port 80 with usernamerstudioand passwordbioc. For the CWL workshop (ONLY), use Port 8080 with usernameubuntuand passwordbioc. This is all documented on the BiocWorkshops2019 repo, along with how to run the workshops via Docker “at home”, a nice feature in my opinion.

Sean Davis (23:27:10) (in thread): > Finally, the bookdown book has had to wait. There are a lot of moving parts and debugging bookdown for each package proved a bridge too far for the conference.

Sean Davis (23:30:05): > Two packages are not yet successfully built and both take quite a while (as long as the workshop in one case) to build. Troubleshooting should be easier in the future with workshop and materials available via docker.

Sean Davis (23:35:00): > Details on how to run via Docker are here:https://github.com/Bioconductor/BiocWorkshops2019/blob/master/RUN_LOCAL.md

Sean Davis (23:36:59): > The download of packages is large (about 5GB, much larger uncompressed). I could have baked all that into the Docker image, but it seems that using a volume makes for a more flexible and portable solution when the packages or materials are changing.

2019-06-19

Lori Shepherd (07:50:42) (in thread): > The configuration of the AMI isn’t working with our course helper that we use to launch all the instances - @nturaga and@Martin Morgan- maybe we could look at this this morning together to maybe figure out how to change the course helper - we would have to document everything careful to revert it back to be able to use it as we normally do after the course

Lori Shepherd (07:53:31) (in thread): > It actually might be okay - but it takes a few minutes for each instance to boot up without ERROR at first - normally we would see this every once and awhile not on every launch …. I’ll try and do some more testing and debugging -

Sean Davis (07:55:45) (in thread): > Is there a timeout built into the course helper?

Lori Shepherd (07:58:40) (in thread): > probably - trying to search the code for it …

Sean Davis (08:09:01) (in thread): > Just so we are clear, a “connection refused” after navigating to the assigned URL is not a problem unless it persists more than 3-4 minutes.

Sean Davis (08:12:51) (in thread): > Finally, the BiocCWL workshop will be on port 8080 with usernameubuntuand passwordbioc.

Lori Shepherd (08:13:19) (in thread): > Yes didn’t realize it would take longer to spin up which was the confusion - It looks like it takes about 2-3 minutes right now for the connection errors to remedy so we should make sure to announce this

Sean Davis (08:13:20) (in thread): > Do you mind if I send the test link to that author to test things?

Lori Shepherd (08:15:13) (in thread): > go for it - I’ll increase the testing limit - I’ll keep it up until tomorrow night

Lori Shepherd (08:16:45) (in thread): > I’m fixing the welcome page to have username rstudio instead of ubuntu now … it should update momenaritly

Sean Davis (08:42:21) (in thread): > Looks great.

Nitesh Turaga (08:58:54): > @Nitesh Turaga has joined the channel

Nitesh Turaga (14:01:35): > This is what shows up on my Rstudio instance launched through the courses website. When I try to dobrowseVigenettes("<package name>")and click on HTML. - File (PNG): Screen Shot 2019-06-19 at 2.00.05 PM.png

Nitesh Turaga (14:03:35): > nvm…apparently i’m supposed to just navigate manually to the vignette document from within Rstudio. Never did that before with Rstudio.

Nitesh Turaga (14:03:39): > thanks@Lori Shepherd

Lori Shepherd (14:04:14): > I think that might be an issue with pops ups and the EC2 instance - all the vignettes are available and display if you navigate through the package pane

Charlotte Soneson (14:19:13) (in thread): > Our RStudio server does the same - butvignette("pkgName")typically works

2019-06-20

Sanjeev Sariya (17:37:06): > @Sanjeev Sariya has joined the channel

Marko Zecevic (19:38:58): > @Marko Zecevic has joined the channel

2019-06-22

Sean Davis (13:29:35): > @Levi Waldron@Marcel Ramos Pérez@Lori ShepherdCould you take a look at website hosting materials for workshops:http://biocworkshops2019.bioconductor.org.s3-website-us-east-1.amazonaws.com

Sean Davis (13:30:57): > Does anyone know if Bioconductor has a public bucket on AWS where we can host this?

2019-06-23

Matt Brauer (23:05:23): > @Matt Brauer has joined the channel

2019-06-24

Levi Waldron (11:38:20) (in thread): > Sorry I missed this message til today Sean. So glad you were able to get this up in time for the conference.

2019-06-25

Burton Karger (06:56:47): > @Burton Karger has joined the channel

Ghada Soliman (09:45:00): > @Ghada Soliman has joined the channel

Andrew McDavid (16:12:04): > @Andrew McDavid has joined the channel

2019-06-26

Junhao Li (13:27:45): > @Junhao Li has joined the channel

2019-08-13

Marcel Ramos Pérez (11:12:07): > Hi Sean,@Sean DavisI want to build the book but I’m getting some errors for some workshops. How did you build the workshops? Do you have a repo with changes that I can apply to build the book? Thanks

2019-08-14

Sean Davis (11:33:31): > The chapters were built using the docker and installed packages used for the workshops. I think that all workshops were taken, at the end, from the original repos.

Sean Davis (11:34:58): > In practice, I’d like to see us take the process to the point that we can build the book from chapters that work and leave the others out, similar to the way the build system works for packages.

Sean Davis (11:37:12): > I hacked blogdown to allow parallel builds (https://github.com/seandavi/parblogdown) as well to facilitate building the website in a reasonable amount of time, but I didn’t do the same with bookdown.

2019-09-05

Rob Amezquita (14:23:39): > @Rob Amezquita has joined the channel

2019-09-09

Kevin Wang (23:15:27): > @Kevin Wang has joined the channel

2019-10-11

Nathan Sheffield (11:08:54): > @Nathan Sheffield has joined the channel

2019-10-13

Sean Davis (11:03:26): > https://seandavi.github.io/post/learning-github-actions/ - Attachment (seandavi(s12)): Experimenting with Github Actions | seandavi(s12) > GitHub actions allow flexible and potentially complicated actions that comprise workflows that respond to events on Github. Continuous integration, messaging Slack, greeting new contributors, deploying applications, and many other templates are ready for customization and integration into any repo.

Sean Davis (11:03:57): > In the resources section of the post is a reference to the ghactions package for R github actions.

2019-10-14

Levi Waldron (15:46:01): > /invite@Lori Shepherd

Simina Boca (15:46:40): > @Simina Boca has joined the channel

Lauren Hsu (15:46:53): > @Lauren Hsu has joined the channel

Mikhail Dozmorov (15:47:31): > @Mikhail Dozmorov has joined the channel

Zhezhen Wang (15:48:52): > @Zhezhen Wang has joined the channel

2019-10-22

Lorena Pantano (17:15:14): > @Sean DavisI am testing the github actions, and I wanted to start now with the code@Marcel Ramos Pérezhas to compile the book. I saw URL when you shared your screen with me, but I forgot to take notes. Can any of you point me to that script so I can test with one single workshop to get one single chapter?

Sean Davis (17:38:13): > https://github.com/Bioconductor/workshopbuilder/blob/master/inst/scripts/workshop.R

Sean Davis (17:38:42): > Thanks for looking into this,@Lorena Pantano

Marcel Ramos Pérez (17:50:54): > Thanks Sean. I’ll work on getting a vignette walk through up and running.

2019-10-23

Lorena Pantano (18:44:45): > I think I can try locally inside a docker from Bioc and see what happens, I think if it works locally inside a docker it will be easy to convert to an action

Sean Davis (21:53:18): > Yep. That sounds like a great test,@Lorena Pantano.

2019-10-24

Lorena Pantano (09:23:40) (in thread): > is there a way to run the script but pointing to a local repository? seeing the code it seems so, but anything specific to change in the installWorkshop funciton?

Marcel Ramos Pérez (10:32:34) (in thread): > Hi Lorena, you can change the defaults in thezzz.Rfile.

Marcel Ramos Pérez (10:34:40) (in thread): > Or you can change usingworkshopbuilder:::.options$set()

Vince Carey (11:43:03): > @Vince Carey has joined the channel

Lorena Pantano (14:58:06) (in thread): > jut thinking that if this is a github action, it needs to install the repo is testing:slightly_smiling_face:so I would try with the option$set. thanks

Sean Davis (17:24:09): > Hi,@Lorena Pantano,@Marcel Ramos Pérez,@Lori Shepherd.@Vince Careyset up a small github workflow (based on GitHub Actions) that uses bioconductor_full docker image to build a package and then save some build artifacts. This looks like a nice proof-of-concept of the capabilities that@Lorena Pantanoand I were thinking about. Additional actions to, for example, build a pkgdown website and push to github pages, build source and binary packages, and perhaps add successfully built workshops to a central repo are all very doable from what Vince has done (in less than 6 hours!).https://github.com/vjcitn/parody/commit/34542cea247a7e2bdb1629570d89ef00811c999f/checks?check_suite_id=280349421

2019-10-25

Vince Carey (02:44:40): > I made a little more progress on actions –https://github.com/vjcitn/ssrch/runs/274453164includes a task that succeeds in installing a package and getting the install log as an artifact. The build task fails because i don’t see how to properly clone a repo. > > Run git clone git@github.com:vjcitn/ssrch.git > Cloning into 'ssrch'... > error: cannot run ssh: No such file or directory > fatal: unable to fork > ##[error]Process completed with exit code 128. > get build log >

Vince Carey (03:50:01): > OK, using git2r we now have the source tarball as an artifact along with logs…https://github.com/vjcitn/ssrch/runs/274519768

Lorena Pantano (12:49:48): > that is great

Lorena Pantano (12:51:36): > I thoughtactions/checkout@v1would do the cloning, and the variable ${GITHUB_WORKSPACE} is where the repository resides.

Vince Carey (13:17:57): > hi@Lorena Pantanothere is nothing authoritative about my YAML for these actions. if you find more economical approaches please let me know or file a pull request. it turns out we need latex support in the container and i am trying TinyTeX … for old Sweave documents this may be inadequate but Rmd seem to be accommodated.

Vince Carey (14:14:32): > I am giving up on handling legacy Rnw in the container for now. Anything for which bioconductor_full + knitr + rmarkdown + BiocStyle is insufficient may fail. This would include ROCnotes.Rnw from ROC, but superficial conversion of that Rnw to Rmd, to include a yaml header, and proper VignetteEngine in DESCRIPTION, allow a build.

Sean Davis (14:39:49): > Perhaps an issue onhttps://github.com/Bioconductor/bioconductor_fullto discuss the LaTeX issue would be warranted.

Sean Davis (14:41:03): > @Lorena Pantano, that is a great observation aboutactions/checkout. How are actions working out for you so far?

Marcel Ramos Pérez (14:46:29) (in thread): > We have scripts to build the bookdown website and push to github pages inBioconductor/BiocWorkshops2019. Shall we automate this step in theworkshopbuilderpackage? Perhaps_deploy.shcan be a GH action?

Sean Davis (14:47:17): > Given the concerted work on github actions going on here, I’d like to suggest that we use our call on Wednesday, October 30 at 12 noon to discuss.@Vince Careyand@Lorena Pantano, would the two of you be OK with highlighting your work then?

Marcel Ramos Pérez (14:48:00) (in thread): > I also usegit2rto clone / pull workshops inworkshopbuilder

Sean Davis (14:48:03): > For those who have not signed up yet, consider doing so.https://github.com/features/actions

Sean Davis (14:48:44): > Actions is still in Beta, but I have not heard of anyone with an active github account getting denied early access.

2019-10-27

Kevin Rue-Albrecht (13:55:24): > Hi@Sean Davis, I’m just wondering if and how I could join the call on Wednesday. Thanks!

Sean Davis (14:06:18): > Anyone who wants to join us is welcome, for sure. We have been using hangouts.

Sean Davis (14:07:59): > Wednesday, October 30 at 12 noon Eastern Time.https://hangouts.google.com/hangouts/_/calendar/c2VhbmRhdmlAZ21haWwuY29t.3dpu45bv8aoadr7j1snsebask5?authuser=0

Kevin Rue-Albrecht (14:08:33): > Cool. I’ve just started reading about Actions this weekend, and I’d be curious to hear how you guys put it in… action (:kissing:)

Kevin Rue-Albrecht (14:09:25) (in thread): > Reading the various GH repos mentioned here is a bit too “static code” for me. I expect I’d catch up a bit faster with the call. Even just being a fly on the wall

Sean Davis (14:09:32): > If you want to use Actions, be sure to sign up for the Beta. They are not yet generally available.

Kevin Rue-Albrecht (14:09:55) (in thread): > Thanks to your earlier message I’ve already done that part

Sean Davis (14:12:20) (in thread): > Let me know what would be helpful for you. We started out as an informal group to work on the workshop book, but I think the charge is morphing a bit to begin to think more generally.

Kevin Rue-Albrecht (14:18:10) (in thread): > I guess at this point, it would be mostly about catching up with where you guys are at, and get inspired if and how I can contribute and learn something in the process. > Around bioc2019 I naively hoped to contribute toward building the book, but then I ran out of time and steam. Right now I’m reading the actions docs, and your “learning-github-actions” page.

Sean Davis (14:39:13) (in thread): > We may want to revisit what we want to accomplish, also. This group has grown, so it will be useful to hear what everyone wants to get out of participating.

Kevin Rue-Albrecht (14:42:34) (in thread): > It might be beside the point here, but reading about Actions makes me wonder whether a relatively useful action could be: > When users submit an issue with a MWE code chunk, to run that code chunk using the latest (release) version of all relevant packages, and display the result as a sort ofreprex. > That could (1) encourage users to submit working MWE code, (2) facilitate a quick preview/instant reply by the developer without having to run/debug the MWE, (3) highlight version issues if users submit code that doesn’t work on their installation but does work with the latest packages.

Kevin Rue-Albrecht (14:42:57) (in thread): > (those last few points are just talking from experience:grin:)

Sean Davis (20:35:57) (in thread): > @Kevin Rue-Albrechtthat sounds like an interesting use case. Are you good with doing a little homework on what that might entail? Looks like this might be helpful:https://help.github.com/en/github/automating-your-workflow-with-github-actions/events-that-trigger-workflows#issues-event-issues

2019-10-28

Kevin Rue-Albrecht (06:47:22) (in thread): > I’m keen to have a look this week, though I can’t promise a timeline: I’ve got annual leave from Friday to mid-November, and tomorrow’s my institute “retreat” day

Sean Davis (07:05:33) (in thread): > No “assignments” or “timelines” here. Just keep us updated on what you learn in your own time.

Sean Davis (14:18:57): > I created a skeletonBiocActionspackage for collecting Actions, related code, and documentation. This needn’t be the only approach or even one of the “right” ones, but I wanted to get something concrete in place to allow collaboration.https://github.com/seandavi/BiocActions

Sean Davis (14:20:45): > Actions are the basic building blocks of workflows and can be shared directly from github without a package checkout if needed. Add new actions by dropping a folder with an appropriate set of files in theinst/actionsdirectory.

Sean Davis (14:21:29): > And an example use: > > on: [push] > > jobs: > hello_world_job: > runs-on: ubuntu-latest > name: A job to say hello > steps: > - name: Hello world action step > id: hello > uses: seandavi/BiocActions/inst/actions/hello-world-docker-action@master > with: > who-to-greet: 'Mona the Octocat' > # Use the output from the `hello` step > - name: Get the output time > run: echo "The time was ${{ steps.hello.outputs.time }}" >

2019-10-30

Kevin Rue-Albrecht (04:23:44): > Hi again. > Unfortunately, I won’t be able to join in the call today (4pm UK time) due to another commitment. > > That said, I’ve done a bit of homework yesterday, and managed to set up both the tutorial Docker action (https://github.com/kevinrue/r-issue-mwe-action/actions) and JavaScript action (https://github.com/kevinrue/run-comment/actions), to teach myself the basic. > > From what I can see, only the GitHub actions JavaScript toolkit (https://github.com/actions/toolkit/tree/master/packages/github) currently provides tools to access the event payload, including the body of issue comments. So it looks like that would be the way to go. > Then, I imagine it would be about parsing the body of the issue comment, extract code chunk(s?), run them, capture thereprexand write a new comment in the issue thread.

Sean Davis (10:09:43): > Nice,@Kevin Rue-Albrecht!

Sean Davis (10:10:58): > Just a reminder that we have a call today at 12 Noon EDT. > - Meeting notes and agenda:https://docs.google.com/document/d/1EKerVh-eWviBrWtr3SEH9LwlYOqZ4rHqbgRWGTjJ2Ik/edit?usp=sharing- Call link (google hangouts):https://hangouts.google.com/hangouts/_/calendar/c2VhbmRhdmlAZ21haWwuY29t.3dpu45bv8aoadr7j1snsebask5?authuser=0 - File (Google Docs): Bioc Workshop Committee working document

Sean Davis (10:11:34): > Feel free to add to or edit agenda.

BJ Stubbs (11:19:11): > @BJ Stubbs has joined the channel

Lorena Pantano (11:51:09): > I have as well a meeting at 12pm, so I cannot join but I have been working on getting the worshopbuild package for a single workshop to compile the book. i think it is almost there, and my next step is to create a docker that gives a path with the local workshop package and create the book, (only with that workshop) to use as a test action in github.

Sean Davis (11:51:56): > Awesome,@Lorena Pantano.

2019-11-04

Lorena Pantano (16:32:35): > @Sean Davisor@Marcel Ramos Pérez, I am having issues building the bookdown for the bioc2019, I am just trying to reproduce this:https://github.com/Bioconductor/BiocWorkshops2019/blob/master/_build.sh#L6with the repository without installing all the rest of the chapters, just the index.Rmd. But I keep having an issue with latex when compile topdf_book, not the others:! Undefined control sequence. l.126 \DeclareUnicodeCharacter. Did you get to compile to PDF_BOOK?

Lorena Pantano (16:32:55): > I can compile to the other formats, but not that one

Lorena Pantano (16:33:22): > This is using the bioconductor/bioconductor_full:devel docker

Marcel Ramos Pérez (16:35:20): > For what workshops are you running it?

Lorena Pantano (17:53:14): > I just want to create the book with the index.Rmd without adding any workshop

Lorena Pantano (17:53:22): > the issue is in the index.Rmd file directly

Lorena Pantano (17:53:54): > I am just getting inside the docker machine, get that repo and run that line of code to create the minimal book, without any workshop to render

Lorena Pantano (17:54:58): > I had to install the bookdown package because it wasn’t in the docker machine, I wonder if it is a new version of the package that is generating this issue

2019-11-05

Oriol Senan (10:46:19): > @Oriol Senan has joined the channel

2019-11-07

Kevin Blighe (11:28:08): > @Kevin Blighe has joined the channel

2019-11-13

Sean Davis (09:45:59): > Hi,<!channel>. We have a call scheduled for today at 12 noon EST. Are there updates anyone wants to give today, or should we just do electronic updates here?

Vince Carey (09:47:57): > OK by me to skip call. I have an “actions” request that has been queued for a day and “cancel run” has no effect. Anyone else seen this? I will comment further on developments in github actions space later on.

Lorena Pantano (09:48:13): > I can put here my updates:

Lorena Pantano (09:52:22): - File (Markdown (raw)): Untitled

Sean Davis (09:58:50): > OK. Let’s cancel the all for today and do updates online.

Sean Davis (09:58:54): > @Vince Carey, which repo are you working in?

Vince Carey (09:59:59): > github.com/vjcitn/biocIBC… FWIW it looks like I can just jump the queue … I started a new job there and it is proceeding. Perhaps the cancellation is effective but not shown.

Sean Davis (10:00:32): > @Lorena Pantano, I’m not surprised that the PDF doesn’t compile. Unfortunately, this setup is pretty sensitive.

Lorena Pantano (10:00:51): > ok

Lorena Pantano (10:01:23): > it is an error of xetex, and some function that it doesn’t recognize, but I couldn’t get the.tex file change to get it running

Sean Davis (10:01:29): > I’m leaning away from working toward a pdf book and more toward a coordinated set of html pages. This is both more modular and doesn’t require maintaining two versions.

Lorena Pantano (10:01:38): > ok

Lorena Pantano (10:01:42): > that’s good

Lori Shepherd (10:30:46) (in thread): > I like this idea a lot

Vince Carey (11:36:27) (in thread): > I am also in favor of it. But what are we losing? At the end of the day, major changes might be needed to get a PDF rendering? Those changes could be the effort for a branch of the repo worked on by those who really want PDF for all or part of the content.

Sean Davis (13:16:24) (in thread): > As far as I can tell, we just lose a combined PDF of the workshops without the book.

Lori Shepherd (13:20:45) (in thread): > Which based on the issues with trying to rebuild last years (can I say that before the year is over) combined book (which may or may not still be an issue? ) - having as many individuals PDFs /htmls as possible seems like a better option that trying to do a combined

2019-11-14

Aedin Culhane (17:05:49) (in thread): > Sounds like a great pragmatic option

Aedin Culhane (17:05:55) (in thread): > Its better to get it out there:wink:

2019-11-17

Oriol Senan (12:01:12): > Dear all, I this is my first year on the BioC Congress organization, and also in the workshop organization. I am following the messages, but I am a little bit lost about where to start or how to help. If there is some task that I can do or some doc or link to learn “the basics” I would appreciate and so I can start helping more effectively.

2019-11-20

Marcel Ramos Pérez (14:30:20): > Thanks@Lorena PantanoLorena for testing workshopbuilder and making it work. It would be good if you could post any issues you encounter onhttps://github.com/Bioconductor/workshopbuilder/issues.

2019-11-21

Kevin Rue-Albrecht (06:49:03): > Quick check: what version of R does the workshop book 2019 require now? For the conference, the requirement was Bioconductor (3.10) and R 3.6. > But now, recent updates toscaterhave created defunct functions that cause errors in our vignette. > I’ve made a GitHub release of the workshop as presented at the conference, but I’m about to update the workshop vignette to run on the new devel (Bioconductor (3.11) and R 4.0) - Attachment: Attachment > Finally updating my installation of R-devel to look into https://travis-ci.org/kevinrue/iSEEWorkshop2019 > > 'calculateQCMetrics' is defunct. > Use 'perCellQCMetrics' instead.

Sean Davis (06:54:59): > Thanks,@Kevin Rue-Albrecht! At this point, we are not planning on updating the workshop materials or software for the Bioc workshops. To use the workshops as they were presented, the docker image and tarball of packages will remain available.

Kevin Rue-Albrecht (06:57:14): > Awesome. Thanks Sean

2019-11-27

Lorena Pantano (10:51:53): > @Marcel Ramos PérezI opened a couple of issues so I could start testing the action with the last bioconductor-full version, let me know if those make sense to do or not. Thanks

Lori Shepherd (11:59:26): > Is there a meeting - nowish - I can’t find the link to sign in?

Marcel Ramos Pérez (12:01:56): > I’m not sure if@Sean Davisis here

Marcel Ramos Pérez (12:03:42): > @Lorena PantanoDo you want to meet about some updates?

Marcel Ramos Pérez (12:04:17): > Otherwise we send updates via email

Lori Shepherd (12:04:58): > I’m a little backlogged so I’m going to assume cancelled - see you all next meeting

Lorena Pantano (13:52:26): > yes, I assumed canceled, I think having that being able to install in latest version of R it was the main issue

Lorena Pantano (13:52:27): > thanks!

2019-12-03

Frederick Tan (15:29:26): > @Frederick Tan has joined the channel

2019-12-11

Christine Choirat (12:08:36): > @Christine Choirat has joined the channel

2019-12-17

Jean Yang (02:40:55): > @Jean Yang has joined the channel

2019-12-18

Shubham Gupta (11:21:52): > @Shubham Gupta has joined the channel

2020-03-28

Levi Waldron (10:28:43): > Hi<!channel>just re-activating the channel and re-posting thehttps://github.com/Bioconductor/workshopbuilderrepo. Who is on board to work on the workshop book this year? I will be able to devote more time than I was able to last year. My goal is to come out of BioC2020 with a full set of workshops that can be compiled in Docker, and a compiled version available online with an index table of contents. A presentable PDF version would be a bonus, as would be an automated continuous integration system.

Levi Waldron (10:30:55) (in thread): > Dear@Oriol Senan, if you’re still game we can probably help get you started now! What is your computational background?

Marcel Ramos Pérez (12:46:11): > :man-raising-hand:I think Sean had mentioned the use of GitHub actions. We can move forward with that idea by providing a template yaml file that uses the Bioconductor docker image (maybe even a pkg collection@Martin Morgan) to test build chapters?@Sean Davis@Lorena Pantano

Martin Morgan (12:46:16): > @Martin Morgan has joined the channel

Levi Waldron (13:04:48) (in thread): > So 1) chapters being individually tested on the contributors’ sites against the Bioconductor image, and 2) pulling successfully buildable snapshots to build the complete “book”

Oriol Senan (13:15:31) (in thread): > Yes I am! And I think that I can be more helpful here. > > To summarize it quickly, I have experience (3-6 years) in R, C++ and Python (developing software, including R packages, and data analysis scripts), and I am familiar (0-2 years) with web programming (Shiny, Python, HTML, CSS, Linux Server, Nginx).

2020-03-31

Koen Van den Berge (15:11:50): > @Koen Van den Berge has joined the channel

2020-04-10

Nitesh Turaga (16:03:03): > Marcel just pointed me tohttps://github.com/tidyverse/dplyr/blob/master/.github/workflows/R-CMD-check.yaml, and the fact that they were able to do this across all three platforms on github is very cool.

2020-04-17

Levi Waldron (12:55:39): > Before we send out the email to workshop contributors, any objection to usinghttps://github.com/waldronlab/MultiAssayWorkshopas the “template” to follow, or other suggestions?@Marcel Ramos Pérez@Sean Davis@Nitesh Turaga@Lori Shepherd@Lorena Pantano

Marcel Ramos Pérez (13:03:21): > We may have to rethink usinginst/vignettesbased on an updated workflow. Thoughts@Sean Davis?

Marcel Ramos Pérez (13:04:39): > Also, IIRC Lorena had worked on a template repository@Lorena Pantanocan you link it here?

Levi Waldron (13:05:54) (in thread): > Oh that’s right!

Levi Waldron (13:07:46): > https://github.com/lpantano/dummychapter1

Sean Davis (14:41:59): > I think we should discuss whether we are aiming for a book or a set of independent workshops. I admit that I have heard strong community opinions both ways about how to proceed.

Levi Waldron (14:52:42): > Can we have both? Independent workshops, and a project that pulls them all those Rmd files together and builds a “book”, even if not a bookdown book?

Levi Waldron (14:54:29): > At least, an index of built versions of all the workshops that makes them easy to navigate.

Levi Waldron (14:55:24): > I like having authors be able to build and test their workshops independently, but also really like having some kind of finished product that organizes them all together in one place.

Kayla Interdonato (16:13:56): > @Kayla Interdonato has joined the channel

Sean Davis (18:42:52): > Let’s assume independent workshops and we can build from there. How do folks feel about each as a pkgdown site, or do you all feel we should develop another mechanism to build and publish each workshop?

Levi Waldron (19:14:15): > That sounds good to me@Sean Davis. If we can build these ourselves and index them in a nice way, ideally with an easy way to pick and choose which workshops to include, it could achieve both modularity (authors building and viewing their own workshops) and the convenience of having everything working and organized in one place.

Levi Waldron (19:14:46): > (but I don’t want to talk over other people who have opinions, it’s awfully quiet in here!)

2020-04-18

Leonardo Collado Torres (23:35:35): > @Leonardo Collado Torres has joined the channel

Leonardo Collado Torres (23:37:49): > Hi, > > I think that this is the best channel for this question, or it could be#bioc2020. Could you remind me what are the resources we should plan for? In particular, I’m concerned about memory. I remember that a step near the end of my recount workshop commands in 2019 crashed some sessions. > > Thanks, > Leo

2020-04-19

Levi Waldron (19:08:51): > I couldn’t find athttps://github.com/Bioconductor/BiocWorkshops2019/blob/master/USING_THE_AMI.mdwhat the AMI specs were last year - do you know offhand@Sean Davis?

Sean Davis (20:03:49): > We should aim for 16GB RAM, 4 vcpus, no more than 40GB disk for data. This helps control costs and ends up looking like most people’s laptop these days. I suspect that with an online meeting, attendance at workshops could be higher than previous years.

Sean Davis (20:04:28): > I am still working on getting a clear picture of what the infrastructure will look like.

2020-04-22

Kelly Eckenrode (09:25:59): > @Kelly Eckenrode has joined the channel

2020-04-24

Anthony (20:08:25): > @Anthony has joined the channel

Anthony (20:12:47): > I have a question related to creating chapters. If the workshop is based on a Bioc package, should the workshop vignette be within that package itself? And for docker, will we be using our own docker images (built on top of bioconductor_docker:devel) with our package dependencies pre-loaded?

2020-04-28

Levi Waldron (07:58:40): > Hi@Anthonysorry I missed your question earlier. The workshop vignette needs to exist in the workshop package, which is separate from the Bioconductor package (although you could put it in your Bioconductor package as well if you want). All workshops will be built on the same Docker image, which may contain additional layers on top of bioconductor_docker:devel. If you need dependencies not present in bioconductor_docker:devel, you can include a Dockerfile in your workshop repo, and we’ll incorporate these into the BioC2020 Docker container.

Levi Waldron (08:00:49) (in thread): > @Sean Davis@Marcel Ramos Pérez@Nitesh Turaga@Lorena PantanoFYI, chime in if you have different ideas

Nitesh Turaga (08:37:14) (in thread): > Just a heads up, the docker image is not yet on R-4.0. This will likely take a little time as we depend on the rocker folks to get their image ready.

Anthony (09:49:38): > Thanks Levi! I imagine the docker image will be very large if it needs dependencies for all workshop packages?

Sean Davis (14:07:01) (in thread): > Thanks,@Nitesh Turaga.

2020-04-29

Levi Waldron (13:32:25): > @Anthonythebioconductor/bioconductor_dockerimage already has system dependencies to install almost all Bioconductor packages, so we would only have to add weird non Bioconductor-related dependencies. It’s a few GB, but actually a surprisingly manageable and convenient image.

Anthony (13:49:46) (in thread): > I see, as long as nobody has conflicting version dependencies.

2020-05-02

Levi Waldron (08:33:46) (in thread): > Yes, that we couldn’t handle. I most workshops will likely be installable on the stockbioconductor/bioconductor_docker:develimage, but I think we’ll have to have a single shared image that we add any other dependencies to. If you’re already at that point, we can get the draft BioC2020 image up soon and start accepting pull requests for it.

Levi Waldron (08:39:44) (in thread): > Hey@Nitesh Turaga, do you think we should have a separate BioC2020 Docker image, or do this as a tag onbioconductor/bioconductor_docker:devel? I think I’d be inclined to have a separate image, maybebioconductor/conferences:bioc2020? A possibility would be to build from a docker/ directory athttps://github.com/Bioconductor/BioC2020.

2020-05-03

Nitesh Turaga (09:37:59) (in thread): > I’m like that idea@Levi Waldron. It seems like it’s a good way for people to find conference specific images easily down the line if it’s under the bioconductor repository in dockerhub.

2020-05-06

Sean Davis (13:04:44) (in thread): > I’ll try to bring some clarity to this, but I think the approach will be to start with Bioconductor_docker:devel and then allow individual workshop presenters/authors to extend with a custom docker file. Thus, there will be one docker image for each workshop.

Anthony (13:08:18) (in thread): > ^ I think that will likely be the most simple route that will certainly work. As long as everyone builds on the same bioc layer, having many docker images for all the workshops should total to a manageable size

Sean Davis (14:52:05) (in thread): > We actually used one docker image last year. It worked fine, but it was not a good developer experience.

2020-05-27

Mikhail Dozmorov (18:41:46) (in thread): > @Sean DavisTo follow up on individual Docker images for each workshop - will they be pre-built? Meaning, all workshop-specific packages installed. Asking because my workshop takes ~35min for Travis to build

Mikhail Dozmorov (18:50:00): > It is time to collate the workshops into one page, like it was done last yearhttp://biocworkshops2019.bioconductor.org.s3-website-us-east-1.amazonaws.com/. Adding Travis buttons will help to see which workshops need attention. Do we have any instructions/template how we should make a single workshop page? Or, we can do it directly on the main Bioc2020 web site - any preference?

Sean Davis (18:57:16) (in thread): > Yes, the images will be pre-built. Workshop submitters will just need to supply a dockerfile. Large files should not be included in the image but downloaded by the participants. I need to write all this up. Apologies.

Sean Davis (19:02:00): > I’ll take care of the workshop page.

2020-05-28

Maria Doyle (00:11:09): > @Maria Doyle has joined the channel

Sean Davis (11:29:31): > Hi, all. I put together a github repo for an example workshop. It is a standard R package with a couple of additional features. The main one is a github actions workflow to build a website (using pkgdown) and to build a docker image (based on Bioconductor_docker:devel) that includes the installed package and dependencies. If folks can take a look and perhaps try it out, that would be great.https://github.com/seandavi/BuildABiocWorkshop2020

Sean Davis (11:30:27): > Some detailed instructions for how to reuse the package for your own repos is at:https://seandavi.github.io/BuildABiocWorkshop2020/articles/HOWTO_BUILD_WORKSHOP.html - Attachment (seandavi.github.io): How To Build A Workshop Package > BuildABiocWorkshop2020

Sean Davis (11:34:16): > If you are impatient and want to see the result, run: > > docker run -e PASSWORD=abc -p 8787:8787 seandavi/buildabiocworkshop2020 > > Then, navigate tohttps://localhost:8787/and login withrstudio:abc(the password from above).

Petr Smirnov (11:36:35): > @Petr Smirnov has joined the channel

Gabriele Sales (11:36:38): > @Gabriele Sales has joined the channel

Kelly Street (11:37:10): > @Kelly Street has joined the channel

Petr Smirnov (11:44:39): > @Sean DavisI am a bit confused regarding docker images: is it required for all workshops or only those that require system dependencies not already on bioconductor_docker:devel?

Sean Davis (11:45:12): > Each workshop will have its own docker image.

Sean Davis (11:46:29): > The workshop-specific docker images will contain the installed workshop package and R package dependencies.

Petr Smirnov (11:47:07): > Ok, perfect! Thank you!

Davide Risso (11:47:23): > @Davide Risso has joined the channel

Dror Berel (11:47:41): > @Dror Berel has joined the channel

Hena Ramay (13:55:16): > @Hena Ramay has joined the channel

Kozo Nishida (14:26:11): > @Kozo Nishida has joined the channel

Peter Hickey (18:52:13): > @Peter Hickey has joined the channel

James MacDonald (20:03:53): > @James MacDonald has joined the channel

James MacDonald (20:05:00): > Thanks@Sean DavisI stumbled upon that today and was wondering if that was how we were supposed to proceed…

2020-05-29

Maria Doyle (09:28:43): > Thanks for the example workshop repo@Sean DavisI tried it out and it worked great:+1:

Sean Davis (09:30:18): > Thanks,@Maria Doyle. Any feedback appreciated! If end up applying it to your own repo, let us know so we can view the results on REAL material.

Maria Doyle (10:07:01): > Oh I did apply it to my repo that’s what I meant I tried:slightly_smiling_face:should I post it here?

Sean Davis (12:21:45): > Sure,@Maria Doyle, if you are willing.

Michael Love (14:45:13): > @Michael Love has joined the channel

Michael Love (14:48:44): > this may not be relevant to others, but I was seeing if I could get travis-ci going myself, and it was timing out bc of the package dependencies. Then i went down a rabbit hole of trying to use binaries, but they are not available for R 4.0. so don’t bother withr_package_binariesand if you time out (for me this was at 50 min), I don’t see a way around it just yet

Charlotte Soneson (14:49:37): > GitHub Actions gives you much more time:slightly_smiling_face:

Charlotte Soneson (14:50:50): > (Also, once the caching kicks in on travis usually things become much faster)

Michael Love (14:52:45): > is this something we could switch to, maybe this was solved last year when we used travis and I’m forgetting how

Charlotte Soneson (14:54:55): > The example workshop above seems to be using GH Actions (under.github/workflows)

Michael Love (14:59:16): > sounds good to me

Charlotte Soneson (15:02:25): > We haven’t yet added this GH Actions workflow to our workshop (was planning to have a look tomorrow), but we set it up separately and it works nicely - happy to assist too if there are general questions on that part.

Koen Van den Berge (16:31:38) (in thread): > Same here.

2020-05-30

Maria Doyle (01:34:19) (in thread): > ok, workshop content is still a work in progress but here’s the pkgdown websitehttps://stemangiola.github.io/bioc_2020_tidytranscriptomics/articles/tidytranscriptomics.html - Attachment (stemangiola.github.io): Tidy Transcriptomics > tidytranscriptomics

Maria Doyle (01:35:00) (in thread): > I also tested the Docker and running the commands in the Rmd installed through that, and all ran as expected

Mikhail Dozmorov (08:23:14): > GH Actions work well, just set up the initial draft of the workshophttps://github.com/mdozmorov/HiCcompareWorkshop. If the workshop package checks without errors, then GH actions will create the pkgdown web site and Docker image. Instructions athttps://github.com/seandavi/BuildABiocWorkshop2020are excellent.

Michael Love (09:20:12): > do you think we’ll skip travis this year?

Mikhail Dozmorov (09:27:25): > Seems like it. GH Actions do more, plus they make Docker images needed for this year

Sean Davis (21:39:44): > Hi,<!channel>. What we’ll need this year is a docker image and a website that contains the workflow package details including any materials you want to make available to your participants. The docker image should not contain large files, generally, so orchestrate your workshop materials to do a transfer from an online source. If there is a need, we can set up a location for the large files to live. The workshop athttps://github.com/seandavi/BuildABiocWorkshop2020is fully functional. I have tried to be specific and thorough about how to adapt it to an existing workshop athttps://seandavi.github.io/BuildABiocWorkshop2020/in “articles” (that are populated directly fromvignettes. If set up, you’ll get GH actions to run, build a pkgdown website, do rcmdcheck::rcmdcheck, and finally produce a docker image that gets pushed to docker hub. - Attachment (seandavi.github.io): BuildABiocWorkshop2020 > This package serves as a template for using github actions to prepare a Bioconductor Workshop package for use. In particular, a pkgdown website and docker image are generated based on workflow settings.

Sean Davis (21:40:45): > If you have an existing workshop package, just copy the .github/workflows/basic_checks.yaml into the root of your package and proceed with the instructions.

2020-05-31

Sean Davis (08:44:52): - Attachment: Attachment > ok, workshop content is still a work in progress but here’s the pkgdown website https://stemangiola.github.io/bioc_2020_tidytranscriptomics/articles/tidytranscriptomics.html

Michael Love (09:19:38): > anyone had success installing scran? I’m getting this in theRun dockerstep on Github: > > Bioconductor version 3.12 (BiocManager 1.30.10), ?BiocManager::install for help > * installing **source** package 'scran' ... > **** using staged installation > > ... > > **** byte-compile and prepare package for lazy loading > Bioconductor version 3.12 (BiocManager 1.30.10), ?BiocManager::install for help > Error in dyn.load(file, DLLpath = DLLpath, ...) : > unable to load shared object '/usr/local/lib/R/site-library/igraph/libs/igraph.so': > libglpk.so.40: cannot open shared object file: No such file or directory > Calls: <Anonymous> ... asNamespace -> loadNamespace -> library.dynam -> dyn.load > Execution halted > ERROR: lazy loading failed for package 'scran' > * removing '/usr/local/lib/R/site-library/scran' > Error in i.p(...) : > (converted from warning) installation of package 'scran' had non-zero exit status > Calls: <Anonymous> ... with_rprofile_user -> with_envvar -> force -> force -> i.p > Execution halted >

Michael Love (09:24:41): > Looks likeigraphwas built as a binary, maybe I will try to force it as a source installation? > > * installing **binary** package 'igraph' ... > * DONE (igraph) >

Charlotte Soneson (09:27:41): > Looks likeglpkmay be missing from the system(?)

Michael Love (09:53:37): > so maybe I could try solving this in this chunk of the yaml? > > # This lets us augment with additional dependencies > - name: Install system dependencies > if: runner.os == 'Linux' > env: > RHUB_PLATFORM: linux-x86_64-ubuntu-gcc > run: | > Rscript -e "remotes::install_github('r-hub/sysreqs')" > sysreqs=$(Rscript -e "cat(sysreqs::sysreq_commands('DESCRIPTION'))") > sudo -s eval "$sysreqs" >

Charlotte Soneson (09:55:04): > Yes, I wonder if adding something like > > sudo apt-get install -y libglpk-dev > > there might help.

Michael Love (10:00:45): > i’ll give it a shot, thanks Charlotte

Sean Davis (13:16:00): > @Michael Loveand@Charlotte Sonesonthe place to address this is in the Dockerfile. The build steps in the YAML will not be executed when building the docker image. > To test locally, you should be able to do: > > docker build . > > inside your package directory. Once that works as expected, then commit that version of the Dockerfile and push. From there, GH actions will build using the new, fixed Dockerfile.

Michael Love (13:39:41): > ok will keep working on it

Sean Davis (18:00:03): > @Nitesh Turaga@Vince CareyAny idea if scran needs something special in the bioconductor_docker:devel?

Vince Carey (18:29:06): > are you having problems building it? i just built from source underus.gcr.io/anvil-gcr-public/anvil-rstudio-bioconductor:0.0.5… looking athttps://github.com/anvilproject/anvil-docker/blob/master/anvil-rstudio-bioconductor/install.Rand the Dockerfile parallel to it, I don’t see any special accommodation.

Vince Carey (18:41:13): > now I see Mike’s issue – since glpk is an optional SystemRequirement of igraph it is conceivable that a found binary package would refer to it … but i do not see glpk in the linux system defined by the bioc container

stefano mangiola (20:19:47): > @stefano mangiola has joined the channel

2020-06-01

Charlotte Soneson (01:41:47): > Manually adding > > RUN apt-get update && apt-get install -y libglpk-dev > > to the Dockerfile seems to have solved the issue for us.

Martin Morgan (05:36:04): > Remember to clean up after apt-get (in the same RUN command) so that the image size is not inflated by layers of cached apt files > > RUN apt-get update && apt-get install -y \ > ... \ > apt-get clean && \ > rm -rf /var/lib/apt/lists/* > > When I install igraph locally (viadocker build .; also if I install scran) it installs without problems from source. As Vince mentioned, libglpk is an optional dependency (in the igraphconfigure.acfile) and I guess the source installation adapts appropriately. How can I trigger a binary installation, locally?

Vince Carey (07:30:33): > Right, it would be nice to know how the offending image of igraph got into the ecosystem that Mike was working with. The setting of options()$repos might be useful.

Michael Love (07:41:44): > I was using the template, I wasnt able to try it out locally yesterday but will try today

Qian Liu (07:53:28): > @Qian Liu has joined the channel

Nitesh Turaga (08:46:31) (in thread): > Hi@Sean DavisIt seems like it needslibglpk-devare you using theRELEASE_3_11version ofdevelversion?

Nitesh Turaga (08:47:31): > So, it seems like the solution is just inherit from the image and add a single library, but no changes to bioconductor_docker itself.

Michael Love (08:53:29): > Thanks everyone for your help

Sean Davis (09:27:34) (in thread): > bioconductor_docker:devel

Sean Davis (09:28:10) (in thread): > We have always used devel for workshops, I think.

John Lawson (11:33:50): > @John Lawson has joined the channel

Levi Waldron (11:44:08) (in thread): > Why not add libglpk-dev to bioconductor_docker? It does provide some additional igraph functionality if I’m reading the docs right, for examplehttps://igraph.org/r/doc/cluster_optimal.html. Even if it’s optional, it’s not huge (“After this operation, 22.8 MB of additional disk space will be used.“) and we’ve seen already two people seeming to need it.

Nitesh Turaga (11:49:15) (in thread): > I’m ok with this too. I will test it real quick and make the change as needed.

Nitesh Turaga (12:22:42) (in thread): > Ok, thank you!

Martin Morgan (13:06:27) (in thread): > I’m interested in knowing how to know it was supposed to be there – igraphdoeswork when installed from source. I’m nervous just adding software without understanding why, and if possible asserting through some testing procedure (like loading the package after installation) that the software is necessary, otherwise we end up x months later wondering why the heck some weird piece of software is present, and so maybe we should just remove it?

Nitesh Turaga (13:08:58) (in thread): > Ok, i’m in agreement with this as well.

Nitesh Turaga (13:10:24) (in thread): > Did this work@Michael Love?

Michael Love (13:13:06) (in thread): > i got stuck yesterday with Docker on my laptop, i needed to make more space and didn’t get a chance to try

Nitesh Turaga (13:13:20) (in thread): > I see, ok.

Nitesh Turaga (13:13:29) (in thread): > thanks!

Michael Love (13:13:30) (in thread): > so now i’m using theapt-getapproach that Martin posted which works for me

Nitesh Turaga (13:14:18) (in thread): > I thinkscranworks just fine when installed from source like Martin mentioned.

Michael Love (13:16:13) (in thread): > ok dumb question, how do i tell github actions to install a particular package from source?

Nitesh Turaga (13:17:34) (in thread): > Not so dumb, but maybe@Charlotte Sonesoncan help with this? I’m not very good with Github Actions:confused:

Charlotte Soneson (13:20:54) (in thread): > Hm, not sure - I guess i would add an explicit line installing the package (withtype = "source")) in the “Install dependencies” part. I’ve never done it to force a source installation, but I did it to force-install some packages that would be installed in the wrong order byremotes.

Michael Love (13:26:16) (in thread): > i may stick with what i’ve got (at least for this week) since it’s working for now

Michael Love (13:26:20) (in thread): > thanks all

Sean Davis (13:29:44) (in thread): > One straightforward approach is to routinely independently build, install, and test each Bioc packages in an “empty” bioc_docker:devel image, sort of the equivalent of installing into vanilla R.

Sean Davis (13:31:34) (in thread): > A second approach, though, is to adopt the sysreqs approach of including system dependencies in the DESCRIPTION file. Someone from R-core or R-libs might have to weigh in here, but that approach seems somewhat valuable even though it is early in development.

Arshi Arora (15:22:24): > @Arshi Arora has joined the channel

James MacDonald (19:18:21): > n

Michael Love (19:35:34): > I think I’m all set now. Bc i had a preexisting repo I didnt use the template. So one thing i had to do (maybe others wont) was explictlygit addmy vignette html made by pkgdown to the /articles dir in gh-pages branch

Michael Love (21:28:44): > question about the title and author blocks in the vignette: so we have these two blocks. the first one creates the html